WHOIGit / ifcbdb

IFCB dashboard
MIT License
8 stars 9 forks source link

retrieve last line of adc only for n_triggers #369

Closed shravani-whoi closed 1 year ago

shravani-whoi commented 1 year ago

Added: postgres Bulk update tqdm progress bar

joefutrelle commented 1 year ago
  1. Use the adc_path property on the Bin model to get the path to the ADC file to read; this will obviate the need for the user to specify a root path (and resolve the issue that more than one root path would need to be specified to handle more than one dataset)
  2. If an ADC file is not found, continue processing the remaining bins
shravani-whoi commented 1 year ago

Tested this locally with minimum data. We need to test this on a big sized data to see how bulk update of 1000 rows at a time is handled by the container.

joefutrelle commented 1 year ago

thanks @shravani-whoi we will need to add tqdm to our requirements

joefutrelle commented 1 year ago

No need for a final stats display, so this LGTM

shravani-whoi commented 1 year ago

Tested this locally with minimum data. We need to test this on a big sized data to see how bulk update of 1000 rows at a time is handled by the container.

image