search

ericleasemorgan / reader

Distant Reader, a tool for using & understanding a corpus
GNU General Public License v2.0
20 stars 7 forks source link

Have txt2ent use multiple slurm nodes #131

Closed dbrower closed 4 years ago

dbrower commented 4 years ago

This splits the list of files into groups, and has each node work on a single group.

I'm not sure how to test this since it involves slurm, and all the cord paths are hardcoded, so I cannot run it on my own data set.

ericleasemorgan commented 4 years ago

Interesting, very! If you want to give it a go, then I believe you have permissions on the machine to execute sbatch ./bin/txt2ent-cord.slurm. Wanna try?

dbrower commented 4 years ago

I'll try it out.

dbrower commented 4 years ago

I think this is ready to go. I'm now working on the underlying python script.

ericleasemorgan commented 4 years ago

Thanks but I'm still trying to be my brain around it.

ericleasemorgan commented 4 years ago

We only have five nodes at our disposal, and as of right now, four of them are in use. When these four nodes die (successfully or unsuccessfully), I will see about merging your pull request and we can give your good work a go. I think we will have to wait about forty-eight hours. Okay?

ericleasemorgan commented 4 years ago

Done, but software is never done, and if it were, then it would be called "hardware". Good luck with the use of multiple nodes to do some of our good work. Please share how well it goes.