Finalize on computing environment to run scrapping pipelines

[prabhushrikant]

We need finalize which enviornment we want to use for running scrapping jobs for various sources and storage location for scrapped data like csv files.

Currenlty we are using google drive but the apis are not straightforward , it will be easier to read directly from

• AWS S3 buckets

• Azure File Storage / Blob storage.

• Airflow tool can be used for designing workflow (pipelines) or we can also do Azure Devops Release pipelines.

• Other options in azure could be

  • Azure Logic Apps (using azure functions)
  • Azure Data Factory
  • Azure Databricks

[prabhushrikant]

we have finalized azure functions as the computing environment for our scrapers. It will use the time triggered functions running on consumption plan (serverless mode). Prototype azure function is still in progress.