Handle scrapingant API errors instead of terminating program

Unfortunately my tmux session only contains 1781 lines and it starts with json already, so I can't really tell where or why it started doing this. Here are the latest lines. Seems like there was an API issue on Scrapingant and fredy didn't catch the error perhaps?:

            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36',            useQueryString: true,                                                                                                                                                                               [0/1781]
            'x-api-key': 'REDACTED',                                                                                                                                                            
            Cookie: 'reese84=3:N701nMUwP7cR0e1tzccZxw==:R5PmKGR1/HaiU5lk66aH3WMIjVdOfLqeXx/boVS7yFtuJA+SLJhhbgedGHmh2w8p0AuCGypwoO7o6AkdVF3nJdeUcCuYtSF+TFzrkLBfZto48550OX6lJLE/rGg1eTb3NgMj5VKnRDb86V92AZVFW7ypoyKx4yheokwBnEfqcFZfrIrPlTQa6Y2JpkLPvZPx6bgrMrXXoWqf6735LQYJ4u3ZtzK6gMAEsvEFSOxU66ztVo6QQMNAUFIrNMiO9YnXIw1kHZu37z9ty/9pQQBDMeOSvhFcQEI5jXtN/DWiBWqg5/TP4qlpwnKbfbxofOiYlf7+kjfs3fobUgrldFgycncXw56DQLx/zRpeO7xnLTlOEbFJ6tqvd2crIxwTPpY3wmERB6P+NDWXNQjswxHjjDmc5pIrTb4LtOJTzJqjHdkGOtUT5RQjPQz/vZT36bwk6ljOrpK4gK/VL+9Oti9yHQ==:kW7yWLLhKwaAx7FDY5OmRCaFpKSiw26NvaHWqUDHsyk='                                                                          
          },                                                                                                
          agent: undefined,                                                                                 
          agents: { http: undefined, https: undefined },                                                                                                                                                                
          auth: undefined,                                                                                                                                                                                              
          hostname: 'api.scrapingant.com',                                                                                                                                                                              
          port: null,                                                                                                                                                                                                   
          nativeProtocols: { 'http:': [Object], 'https:': [Object] },                                                                                                                                                   
          pathname: '/v1/general',                                                                                                                                                                                      
          search: '?url=https%3A%2F%2Fwww.immobilienscout24.de%2FSuche%2Fradius%2Fwohnung-mieten%3Fcenterofsearchaddress%3DHeidelberg%3B%3B%3B%3B%3BBahnstadt%26numberofrooms%3D3.0-%26pricetype%3Drentpermonth%26geocoordinates%3D49.40472%3B8.66334%3B4.0%26sorting%3D2%26enteredFrom%3Dresult_list&proxy_country=il'                                                                                                                         
        },                                                                                                                                                                                                              
        _ended: true,                                                                                                                                                                                                   
        _ending: true,                                                                                                                                                                                                  
        _redirectCount: 0,                                                                                                                                                                                              
        _redirects: [],                                                                                     
        _requestBodyLength: 0,                                                                                                                                                                                          
        _requestBodyBuffers: [],                                                                                                                                                                                        
        _onNativeResponse: [Function (anonymous)],                                                                                                                                                                      
        _currentRequest: [Circular *2],                                                                                                                                                                                 
        _currentUrl: 'https://api.scrapingant.com/v1/general?url=https%3A%2F%2Fwww.immobilienscout24.de%2FSuche%2Fradius%2Fwohnung-mieten%3Fcenterofsearchaddress%3DHeidelberg%3B%3B%3B%3B%3BBahnstadt%26numberofrooms%3D3.0-%26pricetype%3Drentpermonth%26geocoordinates%3D49.40472%3B8.66334%3B4.0%26sorting%3D2%26enteredFrom%3Dresult_list&proxy_country=il',                                                                               
        [Symbol(kCapture)]: false                                                                                                                                                                                       
      },                                                                                                                                                                                                                
      [Symbol(kCapture)]: false,                                                                                                                                                                                        
      [Symbol(kNeedDrain)]: false,                                                                                                                                                                                      
      [Symbol(corked)]: 0,                                                                                                                                                                                              
      [Symbol(kOutHeaders)]: [Object: null prototype] {                                                                                                                                                                 
        accept: [ 'Accept', 'application/json, text/plain, */*' ],                                                                                                                                                      
        'user-agent': [                                                                                     
          'User-Agent',                                                                                                                                                                                                 
          'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36'                                                                                           
        ],                                                                                                                                                                                                              
        usequerystring: [ 'useQueryString', true ],                                                                                                                                                                     
        'x-api-key': [ 'x-api-key', 'REDACTED' ],                                                                                                                                               
        cookie: [                                                                                                                                                                                                       
          'Cookie',                                                                                                                                                                                                     
          'reese84=3:N701nMUwP7cR0e1tzccZxw==:R5PmKGR1/HaiU5lk66aH3WMIjVdOfLqeXx/boVS7yFtuJA+SLJhhbgedGHmh2w8p0AuCGypwoO7o6AkdVF3nJdeUcCuYtSF+TFzrkLBfZto48550OX6lJLE/rGg1eTb3NgMj5VKnRDb86V92AZVFW7ypoyKx4yheokwBnEfqcFZfrIrPlTQa6Y2JpkLPvZPx6bgrMrXXoWqf6735LQYJ4u3ZtzK6gMAEsvEFSOxU66ztVo6QQMNAUFIrNMiO9YnXIw1kHZu37z9ty/9pQQBDMeOSvhFcQEI5jXtN/DWiBWqg5/TP4qlpwnKbfbxofOiYlf7+kjfs3fobUgrldFgycncXw56DQLx/zRpeO7xnLTlOEbFJ6tqvd2crIxwTPpY3wmERB6P+NDWXNQjswxHjjDmc5pIrTb4LtOJTzJqjHdkGOtUT5RQjPQz/vZT36bwk6ljOrpK4gK/VL+9Oti9yHQ==:kW7yWLLhKwaAx7FDY5OmRCaFpKSiw26NvaHWqUDHsyk='                                                                                    
        ],                                                                                                                                                                                                              
        host: [ 'Host', 'api.scrapingant.com' ]                                                                                                                                                                         
      }                                                                                                                                                                                                                 
    },                                                                                                      
    data: {                                                                                                 
      detail: 'Internal server error. Try again later or contact us support@scrapingant.com'                                                                                                                            
    }                                                                                                                                                                                                                   
  },                                                                                                                                                                                                                    
  isAxiosError: true,                                                                                                                                                                                                   
  toJSON: [Function: toJSON]                                                                                                                                                                                            
}                             

error Command failed with exit code 1.                                                                                                                                                                                  
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command

The only modification I did to fredy was add some debug statements to lib/notification/adapter/telegram.js:

exports.send = ({ serviceName, newListings, notificationConfig, jobKey }) => {
  const { token, chatId } = notificationConfig.find((adapter) => adapter.id === 'telegram').fields;

  const bot = new TelegramBot(token);

  let message = `Job: ${jobKey} | Service _${serviceName}_ found _${newListings.length}_ new listings:\n\n`;

  message += newListings.map(
    (o) =>
      `*${shorten(o.title.replace(/\*/g, ''), 45)}*\n` +
      [o.address, o.price, o.size].join(' | ') +
      '\n' +
      `[LINK](${o.link})\n\n`
  );

  console.log(chatId);
  console.log(message);
  console.log(opts);

  return bot.sendMessage(chatId, message, opts);
};

orangecoding / fredy

Handle scrapingant API errors instead of terminating program #25