search

USEPA / EPA_MOVES_Model

Estimating emissions for mobile sources
Other
82 stars 23 forks source link

Running MOVES using a non-default port #75

Open Hanisek opened 3 months ago

Hanisek commented 3 months ago

I am having issues assigning a port to MOVES (other than 3306) - I specified a random port in MySQL.txt under the root folder (in this case it is ). I made sure it was available, and it is the same port that MariaDB was launched (in my.cnf), and the base database was imported. No matter how I change the number of the random port, it seems to be sticking with 13131 (which I really do not understand and this value does not mean anything to me).. Is MOVES by default using 13131 (looks like 13132 is also used) - otherwise any guess what went wrong?

PS I tried two ways of writing MySQL.txt and they generated the same results: port ID only, and port ID with a line break at the end.

Hanisek commented 3 months ago

My MOVES folder and xml files in case they are needed. I removed the subfolders in MOVES (too large to upload) but I kept the other files in the root. I was using the sh file to run under Linux. mm401.zip xml.zip

Hanisek commented 3 months ago

To add to my question - I am trying to run MOVES on a supercomputing cluster, where I noticed that I cannot do that if multiple jobs are assigned to the same node (likely due to the conflict of port). I am just wondering if I need to use the worker feature instead (not sure if that will work though).

danielbizercox commented 3 months ago

The issue that you are describing is not actually with MariaDB's port, but with a port MOVES Main is using. You can only ever have one MOVES Main running on a single computer. The way MOVES enforces this is by binding to port 13131. If port 13131 is already bound by a process, the process that you are launching knows there's already another MOVES Main process running, so it quits with an error message. For more information on MOVES architecture, please see section 5 of the MOVES Overview Report.

Tips For Faster MOVES Runs contains information on how to use MOVES workers. Additionally, it has tips on how to split up your RunSpecs. For example, you can take one large RunSpec and run it on one node / VM / computer, or you can split that large RunSpec into multiple smaller RunSpecs and run those simultaneously on separate nodes / VMs / computers. This will have much better performance, because smaller MOVES runs are faster and they can be run in parallel. For ease of post-processing, you can group the multiple output databases for the separate runs into a single database using Output Grouper Tool.

Hanisek commented 3 months ago

Thanks Daniel. To further clarify - I was not trying to run multiple moves on the same computer but the jobs we submitted could be assigned to multiple cores across the same node (on the supercomputing cluster). I read about the MOVES workers before but I never get to truly understand it.. My question is 1) is there any way I can change the default port that MOVES Main uses (efficiency is not my bottleneck - the ability to run on a different port can really help), and 2) can I essentially have the ability to run multiple instances of MOVES if I use the MOVES workers and the sharedDistributedFolderPath?

Thanks again for your prompt reply!

danielbizercox commented 3 months ago

MOVES will use whatever port you specify in MySQL.txt when connecting to the MariaDB database. Therefore, you will want to make sure that the port in MySQL.txt matches the port in MariaDB's my.ini (Windows) or my.conf (Linux). If the port is incorrectly set, you'll see the following behavior:

However, what you are describing sounds like what MOVES does when you try to run multiple MOVES Main processes at the same time on the same computer (node). MOVES is not designed to be run in this way because the main process is not sandboxed--if you run multiple main processes, they will all be using the same execution database, temporary directory, cache, etc., and the runs will interfere with each other.

To prevent multiple instances of MOVES Main from running at the same time, it binds to port 13131. You'll know that you're running into this issue (having multiple instances of MOVES Main running at the same time) because you'll get this error message: ERROR: MOVES was unable to bind to 13131.

This port is not user configurable. And even if you could change it so that each instance was using a different port, your runs would error out because the instances would interfere with each other. This is why you can only have one instance of MOVES Main running per computer (node).

You can use multiple MOVES Workers per computer (node), but the number of workers is not a bottleneck for most MOVES runs. We generally do not see any improvement beyond 3 workers per onroad run. The sharedDistributedFolderPath can be used if you want to run MOVES Workers on other nodes, but again, this has limited effect.