search

ActoKids / AD440_W19_CloudPracticum

3 stars 1 forks source link

Ofa scaper package is too large to be deployed. #117

Open bmenaz opened 5 years ago

bmenaz commented 5 years ago

In the process of creating Travis CI deployment configurations for several crawler packages, Michael has informed me that ofa scraper is too large to be deployed into Lambda directly, so needs to be hosted on s3 bucket where Lambda would get it from. But, due to fact that Travis deploys folders only and not specific files I had an issue with having to ofa package to be excluded from the Lambda deployment of lambda_function.py and then later to be deployed separately into s3 bucket. Solution would be to host ofa package into its own folder so I can tell Travis CI to pick two different folders for separate deployments.

mrvirus9898 commented 5 years ago

@bmenaz It seems that Travis CI's insistence on targeting folders, and has lead to some confusion. Please use the following paths for Production level deployments:

ShadowSealScraper_prod https://github.com/ActoKids/web-crawler/tree/dev/scripts/lambda/production/ss/target%20package

GoogleCalendarScraper_prod https://github.com/ActoKids/web-crawler/tree/dev/scripts/lambda/production/google/target%20package

OFAScraper_prod https://github.com/ActoKids/web-crawler/tree/dev/scripts/lambda/production/ofa/target%20package

bmenaz commented 5 years ago

Thanks for the folder restructuring. I could specify the specific folders where the package zips are located for Travis CI to pick them up.