search

centreformicrosimulation / SimPaths

SimPaths is an open-source microsimulation framework for life course analysis, developed and maintained by CeMPA at the University of Essex
5 stars 10 forks source link

Experimental/headless setup #37

Closed andrewbaxter439 closed 7 months ago

andrewbaxter439 commented 8 months ago

Hi @pbronka - I was looking up your code for the old labsim implementation of headless single runs. Had a bit of an experiment with it and I think this seems to work to be able to to a headless run of a single run.

After compiling with mvn clean package, the following line does a headless setup phase:

java -jar singlerun.jar -c UK -s 2017 -g false -Setup

This would then allow a script to move on to java -jar multirun.jar ... without needing to do GUI setup bits.

Possible uses of this would be

Seems to be working on my computer. Can check over things robustly to make sure no unintended side effects!

pbronka commented 8 months ago

Hi @andrewbaxter439 - this is certainly useful and I'm quite excited about the possibility of using GitHub Actions for automatic testing. The one thing I was not able to verify with this headless implementation is whether it will run on a machine that has no graphical interface configured at all. Were you able to test in such a setting?

andrewbaxter439 commented 8 months ago

Hi Patryk, got it working on the machine with no display set and it seems to be running smoothly! Cloned the branch and ran:

mvn clean package
java -jar singlerun.jar -c UK -s 2017 -g false -Setup
java -jar multirun.jar -r 100 -p 20000 -n 2 -s 2017 -e 2020 -g false -f

And the whole simpaths multirun got going in a matter of minutes.

Had also tried it on github actions and it had got through all the bits that would normally require a display (https://github.com/andrewbaxter439/SimPaths/actions/runs/6949395610/job/18907413324).

At the moment a full github actions run on training data isn't working until training data is ready - it's the multirun part which is crashing here and not quitting out (because of https://github.com/jasmineRepo/JAS-mine-core/pull/44).

One thing that might need reviewed is that currently, to stop complaints about no displays, I've used if (showGui) to stop the JFrame being created, as FormattedDialogBox.create() is the part where it complains about not having a display: 

String title = "Creating " + Parameters.getTaxDonorInputFileName() + ".csv file";

JFrame csvFrame = null;
if (showGui) csvFrame = FormattedDialogBox.create(title, text, 800, 120, null, false, false, showGui);

System.out.println(title);

The other option would be wrapping FormattedDialogBox.create() in try/catch. This might be more the more formal/professional way of doing it, but the if statement seems simpler. Happy to change if more suitable.

andrewbaxter439 commented 8 months ago

I've added instructions and argument details to README.md 👍

andrewbaxter439 commented 8 months ago

Minor things to fix before potential merging:

justin-ven commented 8 months ago

@andrewbaxter439

Minor things to fix before potential merging:

  • Latest updates have deleted 'EUROMODpolicySchedule.xlsx' and 'DatabaseCountryYear.xlsx' which breaks this headless setup run
  • Are these files to be permanently omitted from all future SimPaths versions @justin-ven? I can see the code in SimPathsStart which could re-create these in a headless setup if needs be?

The deleted Excel files are created each time the database is updated - if you update the database once, then the files should be ignored by subsequent updates. Including them in the git repo meant that they were constantly being updated each time a developer with an alternative database pushed changes, resulting in potential conflicts with the database files residing on local systems.

andrewbaxter439 commented 8 months ago

Thanks @justin-ven - have added the steps in the last two commits which update the new runGUIlessSetup function to create the excel files as well. All works now for this headless setup phase.