Closed zengfung closed 1 year ago
Run selective scraping of data, i.e. if the partition key and row key for a particular data already exist, don't run data scraping. This should reduce the workload of this pipeline and is more scalable.
Run selective scraping of data, i.e. if the partition key and row key for a particular data already exist, don't run data scraping. This should reduce the workload of this pipeline and is more scalable.