Processing fails with Read-only file system

[skymandr]

🐛 Description

The processing fails for two files with

[Errno 30] Read-only file system: '/calibration_data/linearity/tables/channel_2_nrbin_2_ncbinfpga_1_ncbinccd_40.npy

I'm not sure where in the code this error is raised, and why it is a problem that it is a Read-only file system (this is intentional, we should not have to write to this partition).

We should:

• [x] find out where this originates
• [x] figure out why it happens
• [x] fix it :)

🔢 To reproduce

The following files are affected:

ops-payload-level1a-v0.4/2022/12/19/7/MATS_OPS_Level0_VC1_APID100_20221218-130207_20221219-131101.parquet
ops-payload-level1a-v0.4/2022/12/19/8/MATS_OPS_Level0_VC1_APID100_20221218-130207_20221219-131101.parquet

✌🏽 Expected behaviour

No error should be raised because of a read-only filesystem when reading instrument data.

🏆 Solution?

I suspect the easiest solution is to set force_table=False in L1_calibration_functions.get_linearized_image; see comment below. Either that, or explicitly set it to False in the call made in L1_calibrate.L1_calibrate, but I think it is a strange default value, so I prefer the first solution.

[skymandr]

@OleMartinChristensen I'm not sure whose umbrella this falls under. Probably ours at @innosat-mats/molflow , but we may need some help finding out where this happens from you at @innosat-mats/scientific .

[skymandr]

The error seems to originate from https://github.com/innosat-mats/MATS-L1-processing/blob/84d06517eab86ab84c1bcbe8b66d217793560641/src/mats_l1_processing/L1_calibration_functions.py#L343-L355 which calls add_table from database_generation. Why this is triggered for some images and not others, I don't know., but since force_table=True by default, this means it tries to write to disk.

Does the filesystem need to be writeable for these instances..? That is to say, should the default be force_table=True? That is solvable, but it seems unnecessarily complicated for temporary data to write it to disk, and it looks from the code snippet above like it is redundant.

Or is this something missing in the preloaded instrument data, that should be pre-calculated, but isn't yet?

My suggestion is to simply set force_table=False in L1_calibration_functions.get_linearized_image.

[OleMartinChristensen]

dont know if it is still relevant. but FYI: the get table is a precalculated set of non-linearity correction. but it depends on the binning, so for images not using one of the binning settings we defined prior to launch the table will not be called.

Thats why its sometimes called and sometimes not

[skymandr]

It is still relevant. My proposed solution is in #104 .

[OleMartinChristensen]

Let me know how this is going @skymandr I realize the behvious is like this. If table is missing my code will try and create a table for the mode and save it to disk. As a solution right now is just as you say to set force_table to false. This means that if there is no table it will use the inverse model for that mode (and not generate a table). This could results in some very slow processing times for certain batches of images though.

[skymandr]

Let me know how this is going @skymandr I realize the behvious is like this. If table is missing my code will try and create a table for the mode and save it to disk. As a solution right now is just as you say to set force_table to false. This means that if there is no table it will use the inverse model for that mode (and not generate a table). This could results in some very slow processing times for certain batches of images though.

My updated proposal just deactivates if for the Lambda-processing. I don't think that should affect anything you do.