search

ricnogfer / pypsa-in-genomedk

Notes on how to install the PyPSA-Eur framework and to run models based on it in GenomeDK
0 stars 2 forks source link

cluster files + folder for temporary files #1

Open martavp opened 1 year ago

martavp commented 1 year ago

It could be relevant to create a cluster_config + snakemae_cluster files similar to what we do in pypsa-in-prime that helps with the parallelization using different nodes https://github.com/martavp/pypsa-in-prime#5-configure-snakemake

It is also relevant to learn where to write the temporary files (similar to what we do in pypsa-in- prime) https://github.com/martavp/pypsa-in-prime#10-using-scratch-memory-for-the-temporary-directory

ricnogfer commented 1 year ago

@martavp, sounds good and will explore these suggestions soon - thanks for sharing the links!

ricnogfer commented 1 year ago

As a reference, instructions on how to set up temporary directory with scratch memory has been added in commits https://github.com/ricnogfer/pypsa-in-genomedk/commit/ada0507a13e7f145f11c6984f8f87342d06b072f and https://github.com/ricnogfer/pypsa-in-genomedk/commit/552499fbbad2b75c88d5cc6645503ab7c16a6efb.

ricnogfer commented 1 year ago

Concerning the following suggestion:

It could be relevant to create a cluster_config + snakemae_cluster files similar to what we do in pypsa-in-prime that helps with the parallelization using different nodes https://github.com/martavp/pypsa-in-prime#5-configure-snakemake

Although this is a nice feature to have, it might require synchronization effort from us, in particular concerning this file: https://github.com/martavp/pypsa-in-prime/blob/main/PRIME_cluster/Snakefile. Given that the logic in PyPSA-Eur changes at every release, we will then have to check if our Snakefile file is impacted by these changes and, if yes, update this file accordingly. Aside this, a (bigger) issue would be if someone uses a version of PyPSA-Eur that our Snakefile is not "compatible" with.

@martavp: should I still implement this last suggestion? Thanks for confirming!

martavp commented 1 year ago

Hi @ricnogfer ,

I don't see the problem here:

Snakefile will evolve with pypsa-eur, that is fine. We only need to create files 'snakemake_cluster' and 'cluster_config' 'snakemake_cluster' is compatible with every snakefile, it just takes care of allocating memory and passing the different rules from Snakefile that can be solved in parallel to different nodes in the cluster. 'cluster_config' is just a .yaml file with the name of the partitions in the cluster and information on where to save the error and log files. Again, this is compatible with every snakefile