Open s2t2 opened 3 years ago
with ThreadPoolExecutor(max_workers=MAX_THREADS, thread_name_prefix="THREAD") as executor:
for batch in batches:
executor.submit(process_and_save_batch, batch, service)
with ThreadPoolExecutor(max_workers=MAX_THREADS, thread_name_prefix="THREAD") as executor:
futures = [executor.submit(perform, batch, bq_service, bas_service) for batch in batches]
for future in as_completed(futures):
job.counter += future.result()
job.progress_report()
with ThreadPoolExecutor(max_workers=MAX_WORKERS, thread_name_prefix="THREAD") as executor:
futures = [executor.submit(perform, group_name, filtered_df, storage.local_dirpath, tokenize) for group_name, filtered_df in groupby]
for future in as_completed(futures):
result = future.result()
Some students have need for threading in their final project, so the repo should provide some notes about parallel processing, using the concurrent futures module.
References
Threads and Thread Pool Executors:
Locks and Semaphores:
Threading on Heroku
Example:
Use a simpler example than this though.