search

jkcshea / ivmte

An R package for implementing the method in Mogstad, Santos, and Torgovitsky (2018, Econometrica).
GNU General Public License v3.0
18 stars 2 forks source link

Log name conflicts when run in parallel #176

Closed jkcshea closed 4 years ago

jkcshea commented 4 years ago

When using the package in parallel, the console output will fail to be saved at times. This appears to be completely random. Below is an example of this. parallell_ivmte.zip

Initial attempts to address this issue revealed that the temporary log file that sink() was diverting output to would run into name conflicts. This makes sense: multiple processors are trying to create a log file with the same name. This was addressed by first checking whether the file existed, and if so, a new name was generated, and the process was repeated until a valid name is reached. This resolved the issue when the number of iterations in the simulation is small, yet fails when the number of iterations is large.

jkcshea commented 4 years ago

Very simple solution, not sure why I hadn't thought of it before: Simply tag the files with the processor ID. Log file name conflicts have been resolved.

slacouture commented 4 years ago

Hi, I think I am falling into this same issue: when I try to run in parallel the output is randomly failing to be saved. The solution you suggest makes sense but I don't know how to implement this. How do I change the log file name? Can you give me a hand with this please? Thank you!

jkcshea commented 4 years ago

Oh, you shouldn't have to implement anything at all. Could you please post an example of your problem? The examples I have seem to be saving all the results.

slacouture commented 4 years ago

Thanks! You can find the example here mwe_parallel_fail.zip

To be clear. I am running this in Acropolis using different nodes. Thus, I am using doMPI to parallel. I can try it too with doParallel to use different processors in the same node as in your example but it still fails.

Maybe it is a problem of the version? In Acropolis I have the latest release, not the most updated version from GitHub

jkcshea commented 4 years ago

Maybe it is a problem of the version? In Acropolis I have the latest release, not the most updated version from GitHub

Just to make sure, is this referring to ivmte, or doMPI? If it's referring to ivmte, then you should update the version you have on Acropolis. since I only implemented the change with log names 2 days ago.

Just so you know, I update the package on GitHub much more regularly than the version on CRAN (though that due for an update once resolving #181 ). Updating packages on CRAN is slower due to various checks and vetting by package managers.

Let me know if the issue still persists.

slacouture commented 4 years ago

Ugh, it was just a problem of the version of ivmte. Sorry about the false alarm, it wasn't clear to me this was not a change to be made by the user. Things are working now. Thank you for your responsiveness

jkcshea commented 4 years ago

Sure thing, no problem at all. Sorry if the post was confusing. Whenever possible, I try to briefly document the solution before closing an issue. Sometimes old problems pop up again, and it's helpful for me to have a quick reminder of what I tried last.