Singularity/litebird_sim installation error on JSS3

[ebikengo]

Hi, this is Ken EBISAWA at ISAS/JAXA. I am trying to install Singularity/litebird_sim package on JAXA's super-computer JSS3. This is what I did: git clone https://github.com/litebird/litebird_sim litebird_sim

Followed the instruction at https://litebird-sim.readthedocs.io/en/latest/installation.html

% cd singularity/ % ./create-singularity-file.sh 20.04 openmpi % module load singularity <-- this is to enable singularity on JSS3 % which singularity /opt/JX/oss/x86_64/singularity/3.6.4/bin/singularity % singularity build --fakeroot litebird_sim.img Singularity

Then, I got the following errors: test/test_destriper.py::test_destriper FAILED [ 3%] test/test_dipole.py::test_solar_dipole_fit FAILED [ 20%]

I already reported this problem to Maurizio, who says this is due to difference of Python version ( these errors should no happen with v 3.6, but Python version used by the Singularity container is < 3.4).

I just put this post for a record, instead of just e-mailing to Maurizio and developpers. I new version of the package is made, please let me know. I will try it.

[ziotom78]

I tested this on the latest master, and the two tests pass. Here is the result of tail -n 50 of the output, might you please post the same in your case?

Python 3.8.10
+ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

+ python3 -c import litebird_sim as lbs; print('Litebird_sim version: ', lbs.__version__)
TOAST INFO: mpi4py not found- using serial operations only
Litebird_sim version:  0.4.0
INFO:    Adding help info
INFO:    Adding environment to container
INFO:    Adding runscript
INFO:    Adding testscript
INFO:    Running testscript
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-5.4.3, py-1.11.0, pluggy-0.13.1
rootdir: /opt/litebird_sim
collected 64 items

test/test_compress.py .                                                  [  1%]
test/test_destriper.py .                                                 [  3%]
test/test_detectors.py .........                                         [ 17%]
test/test_dipole.py ..                                                   [ 20%]
test/test_io.py .......                                                  [ 31%]
test/test_mapping.py ...                                                 [ 35%]
test/test_mbs.py .                                                       [ 37%]
test/test_mock_imo.py .....                                              [ 45%]
test/test_mpi.py ........                                                [ 57%]
test/test_noise.py .                                                     [ 59%]
test/test_observations.py ..                                             [ 62%]
test/test_quaternions.py .....                                           [ 70%]
test/test_scan_map.py .                                                  [ 71%]
test/test_scanning.py .........                                          [ 85%]
test/test_simulations.py ........                                        [ 98%]
test/test_spacecraft.py .                                                [100%]

=============================== warnings summary ===============================
/usr/local/lib/python3.8/dist-packages/_pytest/stepwise.py:108
  /usr/local/lib/python3.8/dist-packages/_pytest/stepwise.py:108: PytestCacheWarning: cache could not write path /opt/litebird_sim/.pytest_cache/v/cache/stepwise
    self.config.cache.set("cache/stepwise", [])

/usr/local/lib/python3.8/dist-packages/_pytest/cacheprovider.py:366
  /usr/local/lib/python3.8/dist-packages/_pytest/cacheprovider.py:366: PytestCacheWarning: cache could not write path /opt/litebird_sim/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", self.cached_nodeids)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================== 64 passed, 2 warnings in 110.84s (0:01:50) ==================
INFO:    Creating SIF file...
INFO:    Build complete: litebird_sim.img

[ebikengo]

Here it is;

[e578@toki singularity]$ tail -n 50 singularity.txt copying images... [ 55%] images/polarization-direction.svg copying images... [ 61%] images/scanning-strategy-example.png copying images... [ 66%] images/right-handed-coordinates.svg copying images... [ 72%] images/simple-scanning-strategy.png copying images... [ 77%] images/jupiter-angular-distance.svg copying images... [ 83%] images/report_example.png copying images... [ 88%] images/mbs_i.png copying images... [ 94%] images/tutorial-bare-report.png copying images... [100%] images/tutorial-coverage-map.png

copying static files... done copying extra files... done dumping search index in English (code: en)... done dumping object inventory... done build succeeded, 1 warning.

The HTML pages are in build/html. Copying contentui stylesheet/javascript... done make: Leaving directory '/opt/litebird_sim/docs' Running the tests... ============================= test session starts ============================== platform linux -- Python 3.8.10, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /opt/litebird_sim collecting ... collected 64 items

test/test_compress.py::test_rle_compression PASSED [ 1%] test/test_destriper.py::test_destriper FAILED [ 3%] test/test_detectors.py::test_detector_from_dict PASSED [ 4%] test/test_detectors.py::test_detector_from_toml PASSED [ 6%] test/test_detectors.py::test_detector_from_imo PASSED [ 7%] test/test_detectors.py::test_freq_channel_creation PASSED [ 9%] test/test_detectors.py::test_freq_channel_from_imo PASSED [ 10%] test/test_detectors.py::test_freq_channel_noise PASSED [ 12%] test/test_detectors.py::test_instrument_creation PASSED [ 14%] test/test_detectors.py::test_instrument_from_imo PASSED [ 15%] test/test_detectors.py::test_det_list_from_imo PASSED [ 17%] test/test_dipole.py::test_dipole_models PASSED [ 18%] test/test_dipole.py::test_solar_dipole_fit FAILED [ 20%] test/test_io.py::test_write_healpix_map_to_hdu PASSED [ 21%] test/test_io.py::test_write_healpix_map PASSED [ 23%] test/test_io.py::test_write_simple_observation PASSED [ 25%] test/test_io.py::test_write_complex_observation_mjd PASSED [ 26%] test/test_io.py::test_write_complex_observation_no_mjd PASSED [ 28%] test/test_io.py::test_read_complex_observation_mjd PASSED [ 29%] test/test_io.py::test_read_complex_observation_no_mjd PASSED [ 31%] test/test_mapping.py::test_accumulate_map_and_info PASSED [ 32%] test/test_mapping.py::test_make_bin_map_api_simulation PASSED [ 34%] test/test_mapping.py::test_make_bin_map_basic_mpi PASSED [ 35%] test/test_mbs.py::test_mbs [e578@toki singularity]$

[ziotom78]

That's really weird… When test fail, the system should print additional information about why a test failed and which line number included the failure. In this case, it seems that the output was truncated after test/test_mbs.py::test_mbs.

Could you please check that all the output was included?

[ebikengo]

This time, I ran without "tee". Thiis what I got: platform linux -- Python 3.8.10, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /opt/litebird_sim collected 64 items

test/test_compress.py::test_rle_compression PASSED [ 1%] test/test_destriper.py::test_destriper FAILED [ 3%] test/test_detectors.py::test_detector_from_dict PASSED [ 4%] test/test_detectors.py::test_detector_from_toml PASSED [ 6%] test/test_detectors.py::test_detector_from_imo PASSED [ 7%] test/test_detectors.py::test_freq_channel_creation PASSED [ 9%] test/test_detectors.py::test_freq_channel_from_imo PASSED [ 10%] test/test_detectors.py::test_freq_channel_noise PASSED [ 12%] test/test_detectors.py::test_instrument_creation PASSED [ 14%] test/test_detectors.py::test_instrument_from_imo PASSED [ 15%] test/test_detectors.py::test_det_list_from_imo PASSED [ 17%] test/test_dipole.py::test_dipole_models PASSED [ 18%] test/test_dipole.py::test_solar_dipole_fit FAILED [ 20%] test/test_io.py::test_write_healpix_map_to_hdu PASSED [ 21%] test/test_io.py::test_write_healpix_map PASSED [ 23%] test/test_io.py::test_write_simple_observation PASSED [ 25%] test/test_io.py::test_write_complex_observation_mjd PASSED [ 26%] test/test_io.py::test_write_complex_observation_no_mjd PASSED [ 28%] test/test_io.py::test_read_complex_observation_mjd PASSED [ 29%] test/test_io.py::test_read_complex_observation_no_mjd PASSED [ 31%] test/test_mapping.py::test_accumulate_map_and_info PASSED [ 32%] test/test_mapping.py::test_make_bin_map_api_simulation PASSED [ 34%] test/test_mapping.py::test_make_bin_map_basic_mpi PASSED [ 35%] test/test_mbs.py::test_mbs FATAL: While performing build: while running engine: exit status 1 [e578@toki singularity]$ ./

[ziotom78]

Judging from the last message printed by your terminal, it seems that we have two problems here:

• The failing test (FAILED) for the dipole and the destriper
• A FATAL error on the MBS test

The latter unfortunately makes the code crash and prevents it from printing useful output to investigate the former.

I have applied some blind patch to the test for MBS, might you please run a git pull and redo the test? If it still fails, I have added some more options to the Singularity scripts that we might use in that case.

[ebikengo]

Thanks, Maurizo. A problem of JAXA's JSS is that it cannot directly access to git hub! What I did is I git clone on a local machine, and copy do JSS. This is tedious, so I will submit a request to JSS so that it can access git hub directory. In any case, for the time being, I will try git pull on my local machine, and copy to JSS, and redo the test.

[ebikengo]

No good news yet (BTW, I was wrong to have said that git hub is not seen from JSS3. It WAS seen, and no problem).

Somehow mpi gives strange error, I tried without mpi

% ./create-singularity-file.sh 20.04 none % singularity build --fakeroot litebird_sim.img Singularity ... ============================================================================================ test session starts ============================================================================================ platform linux -- Python 3.8.10, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /opt/litebird_sim collected 64 items

test/test_compress.py::test_rle_compression PASSED [ 1%] test/test_destriper.py::test_destriper PASSED [ 3%] test/test_detectors.py::test_detector_from_dict PASSED [ 4%] test/test_detectors.py::test_detector_from_toml PASSED [ 6%] test/test_detectors.py::test_detector_from_imo PASSED [ 7%] test/test_detectors.py::test_freq_channel_creation PASSED [ 9%] test/test_detectors.py::test_freq_channel_from_imo PASSED [ 10%] test/test_detectors.py::test_freq_channel_noise PASSED [ 12%] test/test_detectors.py::test_instrument_creation PASSED [ 14%] test/test_detectors.py::test_instrument_from_imo PASSED [ 15%] test/test_detectors.py::test_det_list_from_imo PASSED [ 17%] test/test_dipole.py::test_dipole_models PASSED [ 18%] test/test_dipole.py::test_solar_dipole_fit FAILED [ 20%] test/test_io.py::test_write_healpix_map_to_hdu PASSED [ 21%] test/test_io.py::test_write_healpix_map PASSED [ 23%] test/test_io.py::test_write_simple_observation PASSED [ 25%] test/test_io.py::test_write_complex_observation_mjd PASSED [ 26%] test/test_io.py::test_write_complex_observation_no_mjd PASSED [ 28%] test/test_io.py::test_read_complex_observation_mjd PASSED [ 29%] test/test_io.py::test_read_complex_observation_no_mjd PASSED [ 31%] test/test_mapping.py::test_accumulate_map_and_info PASSED [ 32%] test/test_mapping.py::test_make_bin_map_api_simulation PASSED [ 34%] test/test_mapping.py::test_make_bin_map_basic_mpi PASSED [ 35%] test/test_mbs.py::test_mbs FATAL: While performing build: while running engine: exit status 1 [e578@toki singularity]$

[ebikengo]

I will attach the entire log. I found the following message in the log. Can't this be a problem?

[2022-04-20 17:48:11,006 WARNING MPI#0000] IMO config file "/root/.config/litebird_imo/imo.toml" not found. --- Logging error --- Traceback (most recent call last): File "/opt/litebird_sim/litebird_sim/imo/imo.py", line 23, in init with CONFIG_FILE_PATH.open("rt") as inpf: File "/usr/lib/python3.8/pathlib.py", line 1222, in open return io.open(self, mode, buffering, encoding, errors, newline, File "/usr/lib/python3.8/pathlib.py", line 1078, in _opener return self._accessor.open(self, flags, mode) FileNotFoundError: [Errno 2] No such file or directory: '/root/.config/litebird_imo/imo.toml'

singularity_nompi.log .

[ziotom78]

Hi @ebikengo , I have tried to reproduce your problem again and again, but with little success.

However, I realized that there might be limitations in the way singularity is supported on HPC machines. (For instance, I believe this is the case for kekcc, which recommends users to build containers on their own machines.)

Have you tried to build the Singularity image (i.e., the litebird_sim.img) on your own laptop and then just copy that file to JSS3? (Of course, to do this you have to install Singularity on your machine; this should not be a big deal if you are running Linux.)