Open larnoldgithub opened 6 months ago
There is no stable version of v0.7.289 yet - so you should not be using it!
Never use a version that says "-live" on the end.
yes it's for test I only wanted to test your fix for the non-interactive installation. It works better, but the new 'other' dir is not yet manage correctly.
Okay well this isnt a good crash but from the log it looks like you didnt add the lbl and other paths to your command. Use --help to see the paths you need to add. It worked for me with the paths. I'll make the error more clear in future.
Okay if you try again now it should work (I didn't realise you were setting datadir
which will automatically make the lbl and other directories).
Note you probably shouldn't mix the two so you have two options:
However it will work now (I tested it and got the following)
- datadir set from cmd (/scratch2/spirou/drs-data/luc_test)
Making dir datadir: /scratch2/spirou/drs-data/luc_test
- DRS_DATA_RAW set from cmd (/scratch2/spirou/drs-data/luc_test/raw)
Making dir DRS_DATA_RAW: /scratch2/spirou/drs-data/luc_test/raw
- DRS_DATA_WORKING set from cmd (/scratch2/spirou/drs-data/luc_test/tmp)
Making dir DRS_DATA_WORKING: /scratch2/spirou/drs-data/luc_test/tmp
- DRS_DATA_REDUC set from cmd (/scratch2/spirou/drs-data/luc_test/red)
Making dir DRS_DATA_REDUC: /scratch2/spirou/drs-data/luc_test/red
- DRS_CALIB_DB set from cmd (/scratch2/spirou/drs-data/luc_test/calib)
Making dir DRS_CALIB_DB: /scratch2/spirou/drs-data/luc_test/calib
- DRS_TELLU_DB set from cmd (/scratch2/spirou/drs-data/luc_test/tellu)
Making dir DRS_TELLU_DB: /scratch2/spirou/drs-data/luc_test/tellu
- DRS_DATA_OUT set from cmd (/scratch2/spirou/drs-data/luc_test/out)
Making dir DRS_DATA_OUT: /scratch2/spirou/drs-data/luc_test/out
- DRS_DATA_ASSETS set from cmd (/scratch2/spirou/drs-data/luc_test/asset)
Making dir DRS_DATA_ASSETS: /scratch2/spirou/drs-data/luc_test/asset
- DRS_DATA_PLOT set from cmd (/scratch2/spirou/drs-data/luc_test/plot)
Making dir DRS_DATA_PLOT: /scratch2/spirou/drs-data/luc_test/plot
- DRS_DATA_RUN set from cmd (/scratch2/spirou/drs-data/luc_test/run)
Making dir DRS_DATA_RUN: /scratch2/spirou/drs-data/luc_test/run
- DRS_DATA_MSG set from cmd (/scratch2/spirou/drs-data/luc_test/msg)
Making dir DRS_DATA_MSG: /scratch2/spirou/drs-data/luc_test/msg
- DRS_DATA_OTHER set from datadir (/scratch2/spirou/drs-data/luc_test/other)
Making dir DRS_DATA_OTHER: /scratch2/spirou/drs-data/luc_test/other
- LBL_PATH set from datadir (/scratch2/spirou/drs-data/luc_test/lbl)
Making dir LBL_PATH: /scratch2/spirou/drs-data/luc_test/lbl
- DRS_PLOT set from cmd (0)
- CLEAN set from cmd (True)
I updated v0.7.289-live
now so this should work for your specific commands.
Note though that as this is a live version I am still working on it and it could break without warning.
Thanks Neil. Shouldn't we have lines added in install_script.sh for lb/ and other/ too ? I'll give the new .289-live a try just to see how it behaves at installation.
non-interactive installation is completed, no errors. lbl/ and other/ are here. I ran setup/install_script.sh .
Thanks Neil. Shouldn't we have lines added in install_script.sh for lb/ and other/ too ? I'll give the new .289-live a try just to see how it behaves at installation.
So the install.sh that is created by apero (after installation) will have that in, but a previous version wont until you update it. This is what I meant be use the --help option to find the command arguments to add to any install script you have.
Here is my install.sh (after a successful install):
>> cat install.sh
python install.py \
--dev \
--name="luc_test" \
--config="/scratch2/spirou/drs-settings/" \
--instrument="SPIROU" \
--datadir="/scratch2/spirou/drs-data/luc_test" \
--rawdir="/scratch2/spirou/drs-data/luc_test/raw" \
--tmpdir="/scratch2/spirou/drs-data/luc_test/tmp" \
--reddir="/scratch2/spirou/drs-data/luc_test/red" \
--outdir="/scratch2/spirou/drs-data/luc_test/out" \
--calibdir="/scratch2/spirou/drs-data/luc_test/calib" \
--telludir="/scratch2/spirou/drs-data/luc_test/tellu" \
--plotdir="/scratch2/spirou/drs-data/luc_test/plot" \
--rundir="/scratch2/spirou/drs-data/luc_test/run" \
--assetsdir="/scratch2/spirou/drs-data/luc_test/asset" \
--logdir="/scratch2/spirou/drs-data/luc_test/msg" \
--otherdir="/scratch2/spirou/drs-data/luc_test/other" \
--lbldir="/scratch2/spirou/drs-data/luc_test/lbl" \
--always_create=True \
--plotmode="0" \
--clean="True" \
--clean_no_warning="True" \
--database_mode=2 \
--database_host="cosmos.astro.umontreal.ca" \
--database_user="spirou" \
--database_pass="Covid19!" \
--database_name="spirou" \
--calibtable="luc_test" \
--tellutable="luc_test" \
--findextable="luc_test" \
--logtable="luc_test" \
--astromtable="luc_test" \
--rejecttable="luc_test" \
--langtable="luc_test"
I worked with Conrad this morning and he also did some corrections/improvements on our side to have apero running in a container. Now it's working, installation and profile setting is entirely non-interactive. All apero-data/testxxx/runlbl,other, etc directories are here. As soon as a .289 version is in stable-test state I'll test the processing of the minidata2.
I'd be interested in the details of the container it might be a good idea for all of us to use something more standardized (regarding the issue #708 with consistent datasets)
For the container, we are using Docker container. I know very little about this technology, but the main point of interset for us is that once the container is created, it can be exported to an other machine (on CANFAR for example :) ) the container is created in 5min after running a setup program called 'dockerfile': the docker file gets the apero code from github, do the installation following the required packages and install_script.sh . Once the container is built and run, you can 'get into' your container and source your (your_test).bash.setup. You get the usual colored startup of apero and everything is working. As far as I understand, no need for conda: all you need is in the container. the container is not really a vm, but some kind of isolated object on your machine where you can run what you want and potentially able to be moved on an other machine. I had to install an editor, nano. editors like emacs aren't available (display issues). you can also edit the ini files from outsite the container and the cp it into the container with a specific docker cp command, but using nano is easier. The processing of the mini-data2 set is running in the container !
@njcuk9999 which file(s) do you want to make the comparison with Montreal ?
I wasn't going to compare them but we could try the lbl rdb file for GL699 first?
As for the container it sounds like we should test "moving it to another machine" and see how similar the results are for mini data. I'd like to try your setup here. If we find out that this is the best way for consistent results we could use it at all institutions!
I sent you the GL699 lbl yesterday evening. With a RMS of 3.00m/s it looks a bit worst than what we got a few month ago (closer to 2.6m/s?)
I'll talk with Conrad to see how we could proceed installing the container on a machine in Montreal or on a VM machine on Canfar. That's the next step now.
it looks like the new dir 'other' is not created next to red/ run/ msg/ etc. ?