hpc pdf grab - Githubissues

VisSieve / main

0 stars 0 forks source link

Open DevinBayly opened 5 days ago

DevinBayly commented 5 days ago

test out our first distributed pdf grab

DevinBayly commented 5 days ago

I believe we can close this, I'm now able to run distributed grabs on duck_db files where the following can be specified

number of nodes to process
number of works for each node
offset from the start of the works list so that we don't redo any pdf grabbing already complete

DevinBayly commented 5 days ago

I need to submit the code to the repo though so perhaps we wait until then to close the issue?