PAX ready for bulk GRID processing?

[rizzoa]

HI Paxers, eventually I succeeded to install PAX on GRID. For the moment is installed only at Nikhef node. I just run a test (a file of a run10 dataset ) which created the default hdf5 output. Let me say anyway that the PAX installation procedure is not optimal for a distributed environment like GRID. I have to deal with some issues (pip installation of tqdm package) and so on. Now I have some requests, first if it is possible to fix the issue Jelle already point on https://github.com/XENON1T/pax/issues/257, it is really annoying this DISPLAY runtime error, in any case the data process on GRID will never involve graphical library. Second, right now I just installed the 4.0.1 version, it this version ok do to some test now, or we should have soon some production version? Could you please update me what is the statues of ROOT output ? About the installation, can be possible to have a conda repository with all the needed software? There are just few packages which are available only via pip (avro-python, tqdm and some other I can check). Having everything in some conda repo will speed up installation.

[tunnell]

Hey Alfio, thanks for looking into this. You may be able to just copy the whole anaconda folder to move it to GRID. We're trying to release the ROOT output at the moment (tried today, maybe tomorrow). The display thing needs to be fixed I agree. We can try to get the ROOT output in a release tomorrow so you can run bigger tests.

Do you have to SSH to a VM for the GRID install? Not sure how it works.

The install instructions are aimed toward end users, I agree. You don't actually have to use anaconda to install pax if there is an easier way, but I still don't think any of the other ways are much easier.

Anyways, I'm going to rename your issue since I think your main point is that you want to know when pax is ready for GRID processing?

[rizzoa]

Hi Chris, copying the Anaconda folder & Co from my local UI (which is the new UI) creating a tar file, was my first thought, but then I realize that this not possible since this python installation put as hashbang my local installation path instead to put #!/usr/bin/env python, and this is something I found really silly. The installation usually is doing simply copying a compiled code( a tar file) via GRID jobusing grid command, but in this case I have to compile locally for the reason I told you, anyway after some debugging I succeeded anyway. Ok to rename this thread

[tunnell]

Will we ever have to install again on grid it is it done for forever? Also, how would I start a job if I wanted using this pax? Thanks again

On 02 Nov 2015, at 17:48, Alfio notifications@github.com wrote:

Hi Chris, copying the Anaconda folder & Co from my local UI (which is the new UI) creating a tar file, was my first thought, but then I realize that this not possible since this python installation put as hashbang my local installation path instead to put #!/usr/bin/env python, and this is something I found really silly. The installation usually is doing simply copying a compiled code( a tar file) via GRID jobusing grid command, but in this case I have to compile locally for the reason I told you, anyway after some debugging I succeeded anyway. Ok to rename this thread

— Reply to this email directly or view it on GitHub.

[rizzoa]

Hi, the only things we have ever to install again is the PAX itself if we have to update it. This can be done easily, after I do git pull locally, create the tar file and copy on disk grid and redo python setup.py develop. I will send the simple shell script to do the pax test on grid, I will send this info on the computing/daq list soon.

[tunnell]

We'll be ready for ROOT on GRID in about a week (beta testing at moment)

[rizzoa]

Hi Chris, whenever we are ready for GRID please let me know. In the meantime I noticed that in the README there is a new part called "Check that no other ROOT is seen" which I guess you wrote. I just read and looks to me weird..., I never had any issue using multiple version of ROOT on my laptop of in any GRID UI I setup so far. The most simple things to do and which so far works well, is just to source the thisroot.(c)sh shell script inside the ROOT bin directory before some compilation/running of any code. What I guess is missing when one issues the "source activate pax" command is the $LD_LIBRARY_PATH environment variable which point to the correct ROOT library path. Anyway is there somewhere a more detailed explanation why you came up with this extreme solution ?

[remenska]

Hi Alfio,

If you know which environment variables to unset when using multiple ROOT installations on your laptop, then it's fine. I guess that's what Chris meant in the paragraph, as he explains below how to check that they are not set.

The conda ROOT installation also does thisroot.(c)sh sourcing behind the scene, only explicitly unsetting LD_LIBRARY_PATH because Anaconda python doesn't play well with this one being set.

I hope this clarifies things a bit.

[rizzoa]

Hi Daniela, well, that paragraph for me is kind of missleading and in part uncorrect, but it does not matter... What I want to point out is that what I wrote before, no LD_LIBRARY_PATH are set when the command "source activate pax" is issued....so it is not true that behind the scene the source of thisroot.sh is done, or it is done not properly ad this cause the problem of library conflict if you have multiple root installion. Hope is it clear!

[tunnell]

I can clarify text. In the beta testing of the ROOT output, we hit many issues of different things linking to different ROOTs than were loading at runtime, resulting in segmentation faults. The safest thing is what I have in the README. The README is a recommendation, but there are many other ways to install it. I do, however, agree with Alfio that I should clarify the text a bit.

[rizzoa]

Hi, just for test, I processed one xe100 raw data event with the current version of pax ((4.1.0) on the GRID. The version of ROOT used was 6 and it finish successfully.

[tunnell]

We narrowed down the memory leak. It would be useful to try processing just run 10 AmBe to see if it fails (and for a larger test). However, we still need a few days before we could reprocess everything I'd say. How does that work for you, Alfio? We're just having some issue with ROOT's memory management.

On Fri, Nov 20, 2015 at 8:56 PM, Alfio notifications@github.com wrote:

Hi, just for test, I processed one xe100 raw data event with the current version of pax ((4.1.0) on the GRID. The version of ROOT used was 6 and it finish successfully.

— Reply to this email directly or view it on GitHub https://github.com/XENON1T/pax/issues/258#issuecomment-158508829.

[rizzoa]

Hi the test I made was really fast, I processed just few events. I could run some complete raw file and check for some memory leak

Alfio

Il 20/11/2015 21:07, Christopher Tunnell ha scritto:

We narrowed down the memory leak. It would be useful to try processing just run 10 AmBe to see if it fails (and for a larger test). However, we still need a few days before we could reprocess everything I'd say. How does that work for you, Alfio? We're just having some issue with ROOT's memory management.

On Fri, Nov 20, 2015 at 8:56 PM, Alfio notifications@github.com wrote:

Hi, just for test, I processed one xe100 raw data event with the current version of pax ((4.1.0) on the GRID. The version of ROOT used was 6 and it finish successfully.

— Reply to this email directly or view it on GitHub https://github.com/XENON1T/pax/issues/258#issuecomment-158508829.

---

Reply to this email directly or view it on GitHub: https://github.com/XENON1T/pax/issues/258#issuecomment-158511534

Dr. Alfio Rizzo Associate Research Scientist Columbia Astrophysics Lab (CAL) 550 West 120th Street 10027 Pupin Hall, MC 5247 New York, NY 10027 Tel: +1 212-854-3098 (Columbia Campus) Tel: +1 914-595-5827 (Nevis Lab) Tel: +39 0862-437-757 (LNGS) alfio.rizzo@columbia.edu edo@astro.columbia.edu http://xenon.astro.columbia.edu/

[tunnell]

That would be helpful, thanks

[rizzoa]

HI, is the default output format of PAX ROOT ? Also, how this --cpus option is supposed to work?

[tunnell]

pax output ROOT by default now.

--cpus shouldn't be used at moment since creates super weird problems sometimes

[rizzoa]

Hi, ok, indeed I tried to use this --cpus option, but it does not work at all. This multithread option will be useful when we will process the DM data on ULITE, for the GRID is not needed at all.

[rizzoa]

Hi, btw, for this ROOT testing people are doing, do you some particular steering file ?

[tunnell]

I don't understand. Most people are just using batch queues at present but I hope we can move soon to official reprocessing.

On 23 Nov 2015, at 17:32, Alfio notifications@github.com wrote:

Hi, btw, for this ROOT testing people are doing, do you some particular steering file ?

— Reply to this email directly or view it on GitHub.

[rizzoa]

what I meant for steering file is what you call config file, so any particular .ini bust just default option....

[tunnell]

Hey Alfio, it should be ready now for prime time. Can you try running it? I've been processing xed files individually then combining them with hadd (as Qing suggested). But I think the answer to this question is yes. Thanks for the help.