Before merge test was done legacy scrape job lambda for Ogre city ONLY for first page cron trigger was disabled
Cron job was getting triggered 00:15 UTC (Riga time UTC+3 01:15) TBC
Details of lambda: region: us-east-1. function name : test-docker-lambda
from task scheduler module local job is triggered at 0:30
schedule.every().day.at("00:30").do(execute_ogre_task)
Email arrives at (02:35AM )
First test failover use local scraper when cloud lambda srape file is not available in S3 - success
Ogre City Apartments for sale from ss.lv webscraperv1.5.4 20240720_0035
Last 3 days
2024-07-18 : TSA [A]: 30 LA TBL [B]: 30 AinB [C]: 26 KLAT A notin B [D]: 4 NewAds, B notin A [E]: 4 RM from LAT
LA TBL rows: 30 RA TBL rows: 1562
LA TBL rows: 30 RA TBL rows: 1562
2024-07-19 : TSA [A]: 30 LA TBL [B]: 30 AinB [C]: 28 KLAT A notin B [D]: 2 NewAds, B notin A [E]: 2 RM from LAT
LA TBL rows: 30 RA TBL rows: 1564
LA TBL rows: 30 RA TBL rows: 1564
After stopping cron trigger for lambda
2024-07-20 : TSA [A]: 30 LA TBL [B]: 30 AinB [C]: 27 KLAT A notin B [D]: 3 NewAds, B notin A [E]: 3 RM from LAT
LA TBL rows: 30 RA TBL rows: 1567
yesterday 66 ads where listed
sleep in local ws is 5 sec between each ad url - this means thatscrape job should have lasted 5x66/60 = 5.5 min
CI stage will last at least 5.5 min + curent times
What will be impacted:
First test failover use local scraper when cloud lambda srape file is not available in S3 - success
Ogre City Apartments for sale from ss.lv webscraperv1.5.4 20240720_0035
Last 3 days 2024-07-18 : TSA [A]: 30 LA TBL [B]: 30 AinB [C]: 26 KLAT A notin B [D]: 4 NewAds, B notin A [E]: 4 RM from LAT LA TBL rows: 30 RA TBL rows: 1562 LA TBL rows: 30 RA TBL rows: 1562 2024-07-19 : TSA [A]: 30 LA TBL [B]: 30 AinB [C]: 28 KLAT A notin B [D]: 2 NewAds, B notin A [E]: 2 RM from LAT LA TBL rows: 30 RA TBL rows: 1564 LA TBL rows: 30 RA TBL rows: 1564 After stopping cron trigger for lambda 2024-07-20 : TSA [A]: 30 LA TBL [B]: 30 AinB [C]: 27 KLAT A notin B [D]: 3 NewAds, B notin A [E]: 3 RM from LAT LA TBL rows: 30 RA TBL rows: 1567
yesterday 66 ads where listed
sleep in local ws is 5 sec between each ad url - this means thatscrape job should have lasted 5x66/60 = 5.5 min
CI stage will last at least 5.5 min + curent times
CD stage will last 5.5 min as well