Slurm utilities

[klamike]

Adds a slurm/ folder with template .sbatch files and Julia scripts for automating dataset generation procedure (adapted from this gist)

[mtanneau]

I would like to have a README here that explains how to setup/execute/process data files. The README should have a list of instructions (with code snippets either in bash or julia) that a user can copy-paste and execute, and they should cover most reasonable use-cases. In particular, anything we do with PGLib should work out of the box.

[klamike]

For setup/executing dataset generation, I can make a README.md file in the slurm/ directory. It will be something along the lines of:

1. Create a folder that will store your dataset.
2. Create a TOML config file in this folder with required keys, like the sample.
3. Run julia --project=. slurm/submit_jobs.jl /path/to/toml

Or did you want me to get rid of submit_jobs.jl and replace it with instructions?

For processing data, I think the current README, which contains Python and Julia examples of reading HDF5 at the bottom, is sufficient. What else did you have in mind?

[mtanneau]

For the README: I just want the steps to execute the data generation. You don't need to re-document the file formats nor how to read them afterwards.

I am a little wary of automatically submitting a lot of jobs. I would prefer we write the .sbatch files, and print what command the user should run. That way people have the opportunity to modify the slurm files before submitting.

[mtanneau]

Thanks @klamike ! Can you separate the code that's related to new DC formulations and standard form, so that this PR only focuses on slurm please? That should also fix the broken CI, which I believe is triggered by errors that have to do with new DC formulations

[klamike]

The DCP PTDF-related changes have been removed from this branch. For some reason some ACOPF tests are failing on GitHub, but they all pass locally for me.

Now this branch has:

• Update config
• Add solve!
• Add SLURM utilities

[mtanneau]

I don't understand why CI is failing... Also good locally. It's probably some issue to do with a more recent version of something, somewhere.

[mtanneau]

Ah, now I can reproduce the failures. I submitted a PR to PGLib () to support more recent versions of PowerModels, and I bumped the Ipopt version. I'm investigating the issue

[mtanneau]

I believe I found the issue. Ipopt complains because we are fixing variables when checking against PowerModels. Doing this effectively removes (many) degrees of freedom, and Ipopt complains that it cannot deal with it. I will push a fix in a sec.