search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
137 stars 25 forks source link

Access denied #88

Open toshaklg opened 3 years ago

toshaklg commented 3 years ago

Hello there! I wonder if anybody had a similar issue and know how it can be fixed.

I use MAJA 4.2.1 from logiciels.cnes.fr, 'Avignon' data downloaded with PEPS and when I make an attempt to run it on the test data by using

(base) anton@kz:~/MAJA_R/bin$ ./startmaja -f folders.txt -t 31TFJ -d 2017-01-01 -e 2017-03-31

i get a permission denied after start of executing the worklplan:

2021-04-20 16:04:00,407 [INFO ] =============This is Start_Maja v4.2.0============== 2021-04-20 16:04:00,412 [INFO ] Detecting input products... 2021-04-20 16:04:00,415 [INFO ] 5 L1C product(s) detected for tile 31TFJ in /home/anton/MAJA_INPUT/31TFJ 2021-04-20 16:04:00,415 [WARNI] No L2A products detected for tile 31TFJ in /home/anton/MAJA_INPUT/31TFJ 2021-04-20 16:04:00,416 [INFO ] Skipping CAMS file detection. 2021-04-20 16:04:00,416 [INFO ] Checking GIPP files 2021-04-20 16:04:00,416 [INFO ] Setting up GIPP folder: /home/anton/MAJA_R/bin/gipp 2021-04-20 16:04:00,422 [INFO ] Searching for DTM 2021-04-20 16:04:00,423 [INFO ] Found DTM: /home/anton/MAJA_R/bin/dtm/S2__TEST_AUX_REFDE2_31TFJ_1001.HDR 2021-04-20 16:04:00,428 [INFO ] GIPP Creation succeeded for SENTINEL2_NATIF 2021-04-20 16:04:00,429 [INFO ] Not enough L1 products available for a BACKWARD mode. Beginning with INIT... 2021-04-20 16:04:00,429 [INFO ] Not enough L1 products available for a BACKWARD mode. Continuing with INIT... 2021-04-20 16:04:00,429 [INFO ] 5 workplan(s) successfully created: DATE | TILE | MODE | L1-PRODUCT | ADDITIONAL INFO 2017-01-03 10:44:32 | 31TFJ | INIT | S2A_MSIL1C_20170103T104432_N0204_R008_T31TFJ_20170103T104428.SAFE | Init mode - No previous L2 2017-01-10 10:34:11 | 31TFJ | NOMINAL | S2A_MSIL1C_20170110T103411_N0204_R108_T31TFJ_20170110T103407.SAFE | L2 from previous 2017-01-13 10:44:01 | 31TFJ | NOMINAL | S2A_MSIL1C_20170113T104401_N0204_R008_T31TFJ_20170113T104402.SAFE | L2 from previous 2017-01-30 10:32:51 | 31TFJ | NOMINAL | S2A_MSIL1C_20170130T103251_N0204_R108_T31TFJ_20170130T103249.SAFE | L2 from previous 2017-03-14 10:40:11 | 31TFJ | INIT | S2A_MSIL1C_20170314T104011_N0204_R008_T31TFJ_20170314T104411.SAFE | Init mode - No previous L2 Press Enter to continue... 2021-04-20 16:04:03,041 [INFO ] Beginning workplan execution. 2021-04-20 16:04:03,042 [INFO ] Executing workplan #1/5 /bin/sh: 1: /home/anton/MAJA_R/bin: Permission denied Traceback (most recent call last): File "/home/anton/work/MAJA_R/lib/python/StartMaja/Start_maja.py", line 555, in s.run() File "/home/anton/work/MAJA_R/lib/python/StartMaja/Start_maja.py", line 502, in run wp.execute(self.maja, self.dtm, self.gipp, self.userconf) File "/home/anton/work/MAJA_R/lib/python/StartMaja/Chain/Workplan.py", line 187, in execute return_code = self.launch_maja(maja, wdir=self.wdir, inputdir=self.input_dir, outdir=self.outdir, conf=conf) File "/home/anton/work/MAJA_R/lib/python/StartMaja/Chain/Workplan.py", line 170, in launch_maja return FileSystem.run_external_app(maja, args, logfile=logfile, skip_error=self.skip_errors) File "/home/anton/work/MAJA_R/lib/python/StartMaja/Common/FileSystem.py", line 188, in run_external_app raise subprocess.CalledProcessError(return_code, cmd) subprocess.CalledProcessError: Command '/home/anton/MAJA_R/bin -w /home/anton/MAJA_R/bin/work/Start_maja_95cab0a3918c533c6277ce8eb3320592/maja_working_directory --input /home/anton/MAJA_R/bin/work/Start_maja_95cab0a3918c533c6277ce8eb3320592 --output /home/anton/MAJA_INPUT/31TFJ --mode L2INIT -ucs /home/anton/work/MAJA_R/lib/python/StartMaja/userconf --TileId 31TFJ --loglevel PROGRESS' returned non-zero exit status 126.

I checked the rights and it looks like they are not the issue, but I have no idea what it could be then. Does anyone have any suggestions?

Thanks!

jerome-colin commented 3 years ago

Hi Anton, Just to be sure: did you checked the permission of the root dir of Maja only ? Does it applies recursively to all sub-directories ? Could you please send us the content of folder.txt ? Thanks, Jerome

toshaklg commented 3 years ago

Oh, right, I totally forgot about posting this info as well, here it is:

  1. Content of folder.txt

    [Maja_Inputs] repWork=./work repGipp=./gipp repMNT=./dtm repL1 =/home/anton/MAJA_INPUT repL2 =/home/anton/MAJA_INPUT exeMaja=/home/anton/MAJA_R/bin repCAMS=/path/to/CAMS [DTM_Creation] repRAW=./dtm/raw repGSW=./dtm/gsw

  2. The input directory has the following structure:

    /home/anton/MAJA_INPUT/31TFJ/S2A_MSIL1C_20170103T104432_N0204_R008_T31TFJ_20170103T104428.SAFE/ ... and more .SAFE folders inside 31TFJ

Regarding permissions, I changed them recursively on "Read and Write" for all, but some systemlinks from MAJA directory threw an error that they cannot be changed, so I left them be.

jerome-colin commented 3 years ago

You don't need to set the write permission to all on the maja directory, but 'x' is mandatory for execution. I strongly suggest you separate your 'jobs' from the root directory of maja. As far as I see it, the dtm and gipp are all created in the 'bin' dir of Maja, which is neither clean nor safe. You may instead create a separated 'maja_jobs' in your home dir where you create your 'dtm' and 'gipp' using startmaja, and point 'repL2' to another directory than 'repL1'. You'd get something like ~/MAJA_INPUT => where you already store your L1C ~/MAJA_OUTPUT => where Maja writes output files ~/MAJA_JOBS => where you prepare your batch runs, dtm, gipp, folder.txt, etc... ~/MAJA_R => where you keep your executable safe with 'rx' for other

To make it even easier, you could add the ../MAJA_R/bin directory in your $PATH variable right from your .bashrc

Hope this helps, Jerome

toshaklg commented 3 years ago

Thanks, I will post the results here as soon as I am done.

petket-5 commented 3 years ago

Hi @toshaklg ,

More certainly, the issue stems from the fact that you need to append /maja to the exeMaja path - You need to reference the full path to the maja executable not just the folder it is located in.

Kind regards, Peter

toshaklg commented 3 years ago

Thanks @jerome-colin and @petket-5, I changed the folders, downloaded DEM that MAJA asked me to do and now it looks like it works both ways - with full path and with ./startmaja. However, I now encounter the following issue:

OSError: [Errno 28] No space left on device: '/tmp/raw_dem_gcn2zdi3'

but I assume it asks for even more free space, though I have about 40 GB on the hard drive...

toshaklg commented 3 years ago

Yeah, actually, it is either "no space error" or "permission denied" error which just deliberately appear one after another on different machines.

jerome-colin commented 3 years ago

This looks similar to an open issue on version 4.2.5 : https://gitlab.orfeo-toolbox.org/maja/maja/-/issues/229

toshaklg commented 3 years ago

Okay, with some additional help I found the issue: in the folders.txt exeMaja should actually point on maja executable, not the folder. Instead of

exeMaja=/home/anton/MAJA_R/bin

I need

exeMaja=/home/anton/MAJA_R/bin/maja

In the readme.md here in the repository, it is a bit controversial info, since folders.txt example is accurate but the description below tells to use a folder, rather than executable for maja. I am not saying that this is wrong, but it definitely took me some time to figure out.

Thanks for quick replies here!

olivierhagolle commented 3 years ago

Sorry for the inaccuracy in the documentation, we'll correct the readme ! Thank you for reporting it ! Olivier

jbrossar commented 3 years ago

Hello @toshaklg , Thanks for using startmaja ! I think /tmp directory is mounted as a tmpfs filesystem, could you try to empty it before running startmaja? Don't hesite to report bug directly on the gitlab (https://gitlab.orfeo-toolbox.org/maja/maja/-/tree/develop), now is our main development platform, Kind regards, Julie