search

satyarth934 / globus_pipeline

0 stars 0 forks source link

Import files and process results in batches #3

Open satyarth934 opened 3 months ago

satyarth934 commented 3 months ago

Check if we can get the entire size of the database that is being imported. If it is larger than a specified MAX_SIZE, we can import/process files in batches.

satyarth934 commented 3 months ago

--batch_mode "filecount" | "filesize"

--batch_size \<count> | \<size in MB>