0.7.289 crash during non-interactive installation

[larnoldgithub]

========================================================================================================================================================================================================================================
Settings for SPIROU
========================================================================================================================================================================================================================================
     - datadir set from cmd (/apero-data/test288)
    Making dir datadir: /apero-data/test288
     - DRS_DATA_RAW set from cmd (/apero-data/test288/raw)
    Making dir DRS_DATA_RAW: /apero-data/test288/raw
     - DRS_DATA_WORKING set from cmd (/apero-data/test288/tmp)
    Making dir DRS_DATA_WORKING: /apero-data/test288/tmp
     - DRS_DATA_REDUC set from cmd (/apero-data/test288/red)
    Making dir DRS_DATA_REDUC: /apero-data/test288/red
     - DRS_CALIB_DB set from cmd (/apero-data/test288/calib)
    Making dir DRS_CALIB_DB: /apero-data/test288/calib
     - DRS_TELLU_DB set from cmd (/apero-data/test288/tellu)
    Making dir DRS_TELLU_DB: /apero-data/test288/tellu
     - DRS_DATA_OUT set from cmd (/apero-data/test288/out)
    Making dir DRS_DATA_OUT: /apero-data/test288/out
     - DRS_DATA_ASSETS set from cmd (/apero-data/test288/asset)
    Making dir DRS_DATA_ASSETS: /apero-data/test288/asset
     - DRS_DATA_PLOT set from cmd (/apero-data/test288/plot)
    Making dir DRS_DATA_PLOT: /apero-data/test288/plot
     - DRS_DATA_RUN set from cmd (/apero-data/test288/run)
    Making dir DRS_DATA_RUN: /apero-data/test288/run
     - DRS_DATA_MSG set from cmd (/apero-data/test288/log)
    Making dir DRS_DATA_MSG: /apero-data/test288/log
     - DRS_DATA_OTHER set from datadir (/apero-data/test288/other)
     - LBL_PATH set from datadir (/apero-data/test288/lbl)
     - DRS_PLOT set from cmd (0)
     - CLEAN set from cmd (True)
========================================================================================================================================================================================================================================
Saved installation parameters to: /opt/apero-drs/config/test288/install.sh

========================================================================================================================================================================================================================================
Installing
========================================================================================================================================================================================================================================

 - Getting binary paths

 - Creating config files

 - Updating config files

 - Creating shell scripts

 - Copying files

Traceback (most recent call last):
  File "/opt/apero-drs/setup/install.py", line 671, in <module>
    main()
  File "/opt/apero-drs/setup/install.py", line 642, in main
    allparams = install.clean_install(params, allparams)
  File "/opt/apero-drs/apero/tools/module/setup/drs_installation.py", line 1056, in clean_install
    toolmod = importlib.import_module(RESET_CODE)
 File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/opt/apero-drs/apero/tools/recipes/bin/apero_reset.py", line 14, in <module>
    from apero.core.core import drs_log
  File "/opt/apero-drs/apero/core/core/drs_log.py", line 678, in <module>
    wlog = Logger()
  File "/opt/apero-drs/apero/core/core/drs_log.py", line 84, in __init__
    self.pin = constants.load()
  File "/opt/apero-drs/apero/core/constants/param_functions.py", line 1474, in load_config
    keys, values, sources, instances = _load_from_file(files, modules)
  File "/opt/apero-drs/apero/core/constants/param_functions.py", line 1898, in _load_from_file
    value = mvalue.validate(fvalues[jt], source=fsources[jt])
  File "/opt/apero-drs/apero/core/constants/constant_functions.py", line 229, in validate
    true_value, source = _validate_value(*vargs, **vkwargs)
  File "/opt/apero-drs/apero/core/constants/constant_functions.py", line 1085, in _validate_value
    true_value = _test_dtype(name, value, dtype, source, quiet=quiet)
  File "/opt/apero-drs/apero/core/constants/constant_functions.py", line 992, in _test_dtype
    raise DrsCodedException('00-003-00010', 'error', targs=eargs,
apero.core.core.drs_exceptions.DrsCodedException: 05:25:01.741-**|constant_functions.py._test_dtype()|E[00-003-00010] Const validate error - Key DRS_DATA_OTHER: Path does not exist '/apero-data/test288/other' 
     Function = constant_functions.py._test_dtype()
spdrs@397254e009f0:/opt/apero-drs/setup$

it looks like the new dir 'other' is not created next to red/ run/ msg/ etc. ?

[njcuk9999]

There is no stable version of v0.7.289 yet - so you should not be using it!

Never use a version that says "-live" on the end.

[larnoldgithub]

yes it's for test I only wanted to test your fix for the non-interactive installation. It works better, but the new 'other' dir is not yet manage correctly.

[njcuk9999]

Okay well this isnt a good crash but from the log it looks like you didnt add the lbl and other paths to your command. Use --help to see the paths you need to add. It worked for me with the paths. I'll make the error more clear in future.

[njcuk9999]

Okay if you try again now it should work (I didn't realise you were setting datadir which will automatically make the lbl and other directories).

Note you probably shouldn't mix the two so you have two options:

1. define datadir (and no red/run/plot/assets etc paths)
2. define all paths (use the --help to find them)

However it will work now (I tested it and got the following)

     - datadir set from cmd (/scratch2/spirou/drs-data/luc_test)
        Making dir datadir: /scratch2/spirou/drs-data/luc_test
     - DRS_DATA_RAW set from cmd (/scratch2/spirou/drs-data/luc_test/raw)
        Making dir DRS_DATA_RAW: /scratch2/spirou/drs-data/luc_test/raw
     - DRS_DATA_WORKING set from cmd (/scratch2/spirou/drs-data/luc_test/tmp)
        Making dir DRS_DATA_WORKING: /scratch2/spirou/drs-data/luc_test/tmp
     - DRS_DATA_REDUC set from cmd (/scratch2/spirou/drs-data/luc_test/red)
        Making dir DRS_DATA_REDUC: /scratch2/spirou/drs-data/luc_test/red
     - DRS_CALIB_DB set from cmd (/scratch2/spirou/drs-data/luc_test/calib)
        Making dir DRS_CALIB_DB: /scratch2/spirou/drs-data/luc_test/calib
     - DRS_TELLU_DB set from cmd (/scratch2/spirou/drs-data/luc_test/tellu)
        Making dir DRS_TELLU_DB: /scratch2/spirou/drs-data/luc_test/tellu
     - DRS_DATA_OUT set from cmd (/scratch2/spirou/drs-data/luc_test/out)
        Making dir DRS_DATA_OUT: /scratch2/spirou/drs-data/luc_test/out
     - DRS_DATA_ASSETS set from cmd (/scratch2/spirou/drs-data/luc_test/asset)
        Making dir DRS_DATA_ASSETS: /scratch2/spirou/drs-data/luc_test/asset
     - DRS_DATA_PLOT set from cmd (/scratch2/spirou/drs-data/luc_test/plot)
        Making dir DRS_DATA_PLOT: /scratch2/spirou/drs-data/luc_test/plot
     - DRS_DATA_RUN set from cmd (/scratch2/spirou/drs-data/luc_test/run)
        Making dir DRS_DATA_RUN: /scratch2/spirou/drs-data/luc_test/run
     - DRS_DATA_MSG set from cmd (/scratch2/spirou/drs-data/luc_test/msg)
        Making dir DRS_DATA_MSG: /scratch2/spirou/drs-data/luc_test/msg
     - DRS_DATA_OTHER set from datadir (/scratch2/spirou/drs-data/luc_test/other)
        Making dir DRS_DATA_OTHER: /scratch2/spirou/drs-data/luc_test/other
     - LBL_PATH set from datadir (/scratch2/spirou/drs-data/luc_test/lbl)
        Making dir LBL_PATH: /scratch2/spirou/drs-data/luc_test/lbl
     - DRS_PLOT set from cmd (0)
     - CLEAN set from cmd (True)

[njcuk9999]

I updated v0.7.289-live now so this should work for your specific commands.

Note though that as this is a live version I am still working on it and it could break without warning.

[larnoldgithub]

Thanks Neil. Shouldn't we have lines added in install_script.sh for lb/ and other/ too ? I'll give the new .289-live a try just to see how it behaves at installation.

[larnoldgithub]

non-interactive installation is completed, no errors. lbl/ and other/ are here. I ran setup/install_script.sh .

[njcuk9999]

Thanks Neil. Shouldn't we have lines added in install_script.sh for lb/ and other/ too ? I'll give the new .289-live a try just to see how it behaves at installation.

So the install.sh that is created by apero (after installation) will have that in, but a previous version wont until you update it. This is what I meant be use the --help option to find the command arguments to add to any install script you have.

Here is my install.sh (after a successful install):

>>   cat install.sh
python install.py          \
          --dev    \
          --name="luc_test"    \
          --config="/scratch2/spirou/drs-settings/"    \
          --instrument="SPIROU"    \
          --datadir="/scratch2/spirou/drs-data/luc_test"    \
          --rawdir="/scratch2/spirou/drs-data/luc_test/raw"    \
          --tmpdir="/scratch2/spirou/drs-data/luc_test/tmp"    \
          --reddir="/scratch2/spirou/drs-data/luc_test/red"    \
          --outdir="/scratch2/spirou/drs-data/luc_test/out"    \
          --calibdir="/scratch2/spirou/drs-data/luc_test/calib"    \
          --telludir="/scratch2/spirou/drs-data/luc_test/tellu"    \
          --plotdir="/scratch2/spirou/drs-data/luc_test/plot"    \
          --rundir="/scratch2/spirou/drs-data/luc_test/run"    \
          --assetsdir="/scratch2/spirou/drs-data/luc_test/asset"    \
          --logdir="/scratch2/spirou/drs-data/luc_test/msg"    \
          --otherdir="/scratch2/spirou/drs-data/luc_test/other"    \
          --lbldir="/scratch2/spirou/drs-data/luc_test/lbl"    \
          --always_create=True    \
          --plotmode="0"    \
          --clean="True"    \
          --clean_no_warning="True"    \
          --database_mode=2    \
          --database_host="cosmos.astro.umontreal.ca"    \
          --database_user="spirou"    \
          --database_pass="Covid19!"    \
          --database_name="spirou"    \
          --calibtable="luc_test"    \
          --tellutable="luc_test"    \
          --findextable="luc_test"    \
          --logtable="luc_test"    \
          --astromtable="luc_test"    \
          --rejecttable="luc_test"    \
          --langtable="luc_test"

[larnoldgithub]

I worked with Conrad this morning and he also did some corrections/improvements on our side to have apero running in a container. Now it's working, installation and profile setting is entirely non-interactive. All apero-data/testxxx/runlbl,other, etc directories are here. As soon as a .289 version is in stable-test state I'll test the processing of the minidata2.

[njcuk9999]

I'd be interested in the details of the container it might be a good idea for all of us to use something more standardized (regarding the issue #708 with consistent datasets)

[larnoldgithub]

For the container, we are using Docker container. I know very little about this technology, but the main point of interset for us is that once the container is created, it can be exported to an other machine (on CANFAR for example :) ) the container is created in 5min after running a setup program called 'dockerfile': the docker file gets the apero code from github, do the installation following the required packages and install_script.sh . Once the container is built and run, you can 'get into' your container and source your (your_test).bash.setup. You get the usual colored startup of apero and everything is working. As far as I understand, no need for conda: all you need is in the container. the container is not really a vm, but some kind of isolated object on your machine where you can run what you want and potentially able to be moved on an other machine. I had to install an editor, nano. editors like emacs aren't available (display issues). you can also edit the ini files from outsite the container and the cp it into the container with a specific docker cp command, but using nano is easier. The processing of the mini-data2 set is running in the container !

[larnoldgithub]

@njcuk9999 which file(s) do you want to make the comparison with Montreal ?

[njcuk9999]

I wasn't going to compare them but we could try the lbl rdb file for GL699 first?

As for the container it sounds like we should test "moving it to another machine" and see how similar the results are for mini data. I'd like to try your setup here. If we find out that this is the best way for consistent results we could use it at all institutions!

[larnoldgithub]

I sent you the GL699 lbl yesterday evening. With a RMS of 3.00m/s it looks a bit worst than what we got a few month ago (closer to 2.6m/s?)

I'll talk with Conrad to see how we could proceed installing the container on a machine in Montreal or on a VM machine on Canfar. That's the next step now.