search

Sen2Agri / Sen2Agri-System

Sentinel-2 for Agriculture (Sen2Agri) is a software system processing high resolution satellite images for agricultural purposes funded by ESA (European Space Agency). Please register on the Sen2Agri webpage for Sen2Agri system updates and information.
http://www.esa-sen2agri.org
Other
115 stars 38 forks source link

Unable to read metadata (l3a,l3b) #28

Open smfcrts opened 5 years ago

smfcrts commented 5 years ago

Hello, I run L3A composite through command line: /usr/bin/composite_processing.py --syntdate 20190901 --input "/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190813T112121_N0208_R037_T29SMS_20190813T114617.SAFE/S2A_OPER_SSC_L2VALD_29SMS20190813.HDR " "/mnt/archive/maccs_def/morocco/l2a/S2B_MSIL2A_20190818T112119_N0208_R037_T29SMS_20190818T122949.SAFE/S2B_OPER_SSC_L2VALD_29SMS____20190818.HDR" "/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190823T112121_N0208_R037_T29SMS_20190823T114631.SAFE/S2A_OPER_SSC_L2VALD_29SMS20190823.HDR" "/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190902T112111_N0208_R037_T29SMS_20190902T132637.SAFE/S2A_OPER_SSC_L2VALD_29SMS20190902.HDR" "/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190912T112111_N0208_R037_T29SMS_20190912T114646.SAFE/S2A_OPER_SSC_L2VALD_29SMS____20190912.HDR" "/mnt/archive/maccs_def/morocco/l2a/S2B_MSIL2A_20190917T112119_N0208_R037_T29SMS_20190917T132014.SAFE/S2B_OPER_SSC_L2VALD_29SMS20190917.HDR" "/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190922T112111_N0208_R037_T29SMS_20190922T114652.SAFE/S2A_OPER_SSC_L2VALD_29SMS20190922.HDR" --res 10 --outdir "/mnt/archive/temp/20190901/29SMS/" --bands /usr/share/sen2agri/bands_mapping_s2_L8.txt --scatteringcoe /usr/share/sen2agri/scattering_coeffs_10m.txt --synthalf 25 It gave this error: **The following error occurred during application execution : /home/cudroiu/sen2agri/sen2agri-processors/ProductReaders/MetadataHelpers/src/MetadataHelperFactory.cpp:43: itk::ERROR: MetadataHelperFactory(0x16a8810): Unable to read metadata from /mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190813T112121_N0208_R037_T29SMS_20190813T114617.SAFE/S2A_OPER_SSC_L2VALD_29SMS20190813.HDR OTB app finished in: 0:00:04.178794 OTB application error** Please how can i overcome this issue

smfcrts commented 5 years ago

@lnicola please could you give a solution to this issue

lnicola commented 5 years ago

@smfcrts sorry for the late reply. Can you tell us what version of the system you are running and maybe attach a copy of that HDR file?

smfcrts commented 5 years ago

Hello, The version of the system is 2.0.1, and find below example of the HDR file(I changed here S2A_OPER_SSC_L2VALD_29SMS____20190813.txt the extension to .txt because i can't upload HDR file): [Uploading S2A_OPER_SSC_L2VALD_29SMS____20190813.txt…]()

smfcrts commented 5 years ago

@lnicola I sent a copy of attached HDR file,please have a look on it

lnicola commented 5 years ago

I'll think about it when I find some time, but I didn't see anything obviously wrong there.

smfcrts commented 5 years ago

Thank you @lnicola

smfcrts commented 4 years ago

@lnicola Hi, did you have a look on the HDR file, because i still have the same problem even if the L2A products are well generated.

lnicola commented 4 years ago

I couldn't see anything obviously wrong with the file. Can you upload the whole product somewhere (e.g. https://send.firefox.com) so I can pass it to my colleague who knows more about the L3A?

In the meanwhile, can you test with fewer input products in that command line? Even a single one, if it allows you. Also, what happens if you remove the first product from that list? Do you get an error on the second one as well?

smfcrts commented 4 years ago

I tested with only one product and it give the same error, and even if i remove the first one it give error on the second one. So which product should i upload?

lnicola commented 4 years ago

Either of them, I suppose.

smfcrts commented 4 years ago

Do you think it could be a problem of permission?

lnicola commented 4 years ago

It might be. Does cat /mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190813T112121_N0208_R037_T29SMS_20190813T114617.SAFE/S2A_OPER_SSC_L2VALD_29SMS____20190813.HDR work from the account you're using to run the processor?

smfcrts commented 4 years ago

ok i'll try, what does cat do?

lnicola commented 4 years ago

It should display the contents of that file (it's an XML) to the console/screen. If you get a "Permission denied" error, it means it's not working.

smfcrts commented 4 years ago

yes it works.

lnicola commented 4 years ago

Can you also try this?

$ sudo yum -y install strace $ strace -f /usr/bin/composite_processing.py --syntdate 20190901 --input [...] 2>&1 | tee log.txt # use the whole command line here

When it finishes, log.txt should contain the complete list of operating system calls done by the programs. If there's an OS error, it should show up in there.

PS: I just noticed that your original command line has this:

"/mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190813T112121_N0208_R037_T29SMS_20190813T114617.SAFE/S2A_OPER_SSC_L2VALD_29SMS____20190813.HDR "

Note the extra space before the closing quote. Assuming the file doesn't have a space in its name, you'll get a file not found error on it.

On the other hand, you said you got the same error even after removing the first file, so...

smfcrts commented 4 years ago

the error mention that it couldn't read file, not the file doesn't exit.So I'll try right now what you suggest.

lnicola commented 4 years ago

the error mention that it couldn't read file, not the file doesn't exit.So I'll try right now what you suggest.

I'm pretty sure it's the same thing (it can't read a file that doesn't exist).

smfcrts commented 4 years ago

I couldn't find the log file, it show information on terminal

lnicola commented 4 years ago

There should be a log.txt file in the current directory (from the 2>&1 | tee log.txt part of the command.

smfcrts commented 4 years ago

PLease find attached the log file log.txt

lnicola commented 4 years ago

The file you've attached is truncated:

"log.txt" 2394L, 182930C

It looks like you've opened the file with a text editor (vi?), then copied the screen contents and pasted it to this file. The original file had 2394 lines and 180 KB, while the one above has 38 lines.

smfcrts commented 4 years ago

ah yes you're right

smfcrts commented 4 years ago

because i run the command on sen2agri-service user, so how can i get the log file without using vim.

smfcrts commented 4 years ago

Here is the file but i run the command line with user log.txt

lnicola commented 4 years ago

because i run the command on sen2agri-service user, so how can i get the log file without using vim.

You can copy it to your account with e.g. sudo cp ~sen2agri-service/log.txt ..

Here is the file but i run the command line with user log.txt

From the log: composite_processing.py: error: argument --synthalf is required

smfcrts commented 4 years ago

HI, I added synthalf to the command line.Please find attached the log.txt file log.txt

lnicola commented 4 years ago

error: argument --outdir is required

Can you test it beforehand (without strace) to make sure that the command works?

smfcrts commented 4 years ago

here is the log file without strace log.txt

lnicola commented 4 years ago

All right, at least it's starting now. Can you try with strace? Maybe use strace -s256 -f composite_processing.py [...] this time, it will print the whole paths.

smfcrts commented 4 years ago

Here is the log file with strace -s256 -f log.txt

lnicola commented 4 years ago

So it's not a missing file nor a permission issue. Can you make an archive of the product and upload it somewhere like https://send.firefox.com?

smfcrts commented 4 years ago

I was not able to upload the zip file through the link you sent

lnicola commented 4 years ago

Is it larger than 1 GB? If so, you can make an account there, or try a different service.

smfcrts commented 4 years ago

I do, i created an account

smfcrts commented 4 years ago

https://send.firefox.com/download/411db81e23a5e49c/#wtTGt5QzxrbJ9Ay8Muc9WQ

smfcrts commented 4 years ago

HI @lnicola , please did you have a look on the link

lnicola commented 4 years ago

I was able to reproduce the failure, and it seems to be related to the format of the product metadata. Specifically, we're now using a version of MAJA that outputs a different product format. If you look at https://logiciels.cnes.fr/fr/node/57?type=desc, you'll see two versions, "TM" and "noTM". We use the former, but you might have the other one.

Your product is similar to MACCS (the predecessor of MAJA) products and we can actually open it, but we reject it because of an over-eager validation. You can test it by opening the product HDR file and changing

        <Source>
            <System>MAJA</System>
            <Creator>MAJA_L2_NOMINAL_CHAIN</Creator>
            <Creator_Version>3.2.2</Creator_Version>
            <Creation_Date>UTC=2019-11-04T00:57:14</Creation_Date>
        </Source>

to

        <Source>
            <System>MACCS</System>
            <Creator>MAJA_L2_NOMINAL_CHAIN</Creator>
            <Creator_Version>3.2.2</Creator_Version>
            <Creation_Date>UTC=2019-11-04T00:57:14</Creation_Date>
        </Source>

After making the change the processing started, at least. I couldn't let it finish because of unrelated reasons.

We will probably fix the validation in a future version, but it might be easiest for you to just change the string in the HDR files. This can be done in an automated way. I can guide you through the process, if you want to take that route.

smfcrts commented 4 years ago

Hi, what do you mean by the former one for maja output, it means you use the version TM or noTM. So i have to change MAJA by MACCS? That 's it Le jeu. 16 janv. 2020 à 17:09, Laurențiu Nicola notifications@github.com a écrit :

I was able to reproduce the failure, and it seems to be related to the format of the product metadata. Specifically, we're now using a version of MAJA that outputs a different product format. If you look at https://logiciels.cnes.fr/fr/node/57?type=desc, you'll see two versions, "TM" and "noTM". We use the former, but you might have the other one.

Your product is similar to MACCS (the predecessor of MAJA) products and we can actually open it, but we reject it because of an over-eager validation. You can test it by opening the product HDR file and changing

    <Source>
        <System>MAJA</System>
        <Creator>MAJA_L2_NOMINAL_CHAIN</Creator>
        <Creator_Version>3.2.2</Creator_Version>
        <Creation_Date>UTC=2019-11-04T00:57:14</Creation_Date>
    </Source>

to

    <Source>
        <System>MACCS</System>
        <Creator>MAJA_L2_NOMINAL_CHAIN</Creator>
        <Creator_Version>3.2.2</Creator_Version>
        <Creation_Date>UTC=2019-11-04T00:57:14</Creation_Date>
    </Source>

After making the change, the processing start, at least.

We will probably fix the validation in a future version, but it might be easiest for you to just change the string in the HDR files. This can be done in an automated way. I can guide you through the process, if you want to take that route.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Sen2Agri/Sen2Agri-System/issues/28?email_source=notifications&email_token=ALRSPNN5FZ6MOZQEZTJWVC3Q6CBDPA5CNFSM4JLJJRQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJETPEA#issuecomment-575223696, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALRSPNOX4IBDVXHOVBROQBTQ6CBDPANCNFSM4JLJJRQQ .

lnicola commented 4 years ago

Hi, what do you mean by the former one for maja output, it means you use the version TM or noTM.

We use TM, I believe you have noTM.

So i have to change MAJA by MACCS? That 's it

Yes, in the System element. The Creator doesn't matter.

smfcrts commented 4 years ago

So i change maja to TM, it will solve the problem

Le jeu. 16 janv. 2020 à 22:29, Laurențiu Nicola notifications@github.com a écrit :

Hi, what do you mean by the former one for maja output, it means you use the version TM or noTM.

We use TM, I believe you have noTM.

So i have to change MAJA by MACCS? That 's it

Yes, in the System element. The Creator doesn't matter.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Sen2Agri/Sen2Agri-System/issues/28?email_source=notifications&email_token=ALRSPNJSSQATUS7OCIYCIRLQ6DGTVA5CNFSM4JLJJRQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJFTWIA#issuecomment-575355680, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALRSPNLWANNX3572KEINR33Q6DGTVANCNFSM4JLJJRQQ .

lnicola commented 4 years ago

It should, but you might as well replace that string in the HDR files to keep the L2A products that you already have.

smfcrts commented 4 years ago

yes you're right, so how can i do it automatically

Le jeu. 16 janv. 2020 à 22:36, Laurențiu Nicola notifications@github.com a écrit :

It should, but you might as well replace that string in the HDR files to keep the L2A products that you already have.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Sen2Agri/Sen2Agri-System/issues/28?email_source=notifications&email_token=ALRSPNIIVCYRKNXUWVMJE53Q6DHMBA5CNFSM4JLJJRQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJFUJVQ#issuecomment-575358166, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALRSPNM3ZVDFG3X7UDUQBB3Q6DHMBANCNFSM4JLJJRQQ .

lnicola commented 4 years ago

Not really tested, but something like:

$ sudo su -l sen2agri-service
$ find /mnt/archive/maccs_def/morocco/l2a/ -maxdepth 1 -name "*.HDR" -print0 | xargs -0 sed -i 's|<System>MAJA</System>|<System>MACCS</System>|'

You might need -maxdepth 2 instead of 1.

smfcrts commented 4 years ago

ok i'll test it and i'll let you know, thank you

Le jeu. 16 janv. 2020 à 22:44, Laurențiu Nicola notifications@github.com a écrit :

Not really tested, but something like:

$ sudo su -l sen2agri-service $ find /mnt/archive/maccs_def/morocco/l2a/ -maxdepth 1 -name "*.HDR" -print0 | xargs -0 sed -i 's|MAJA|MACCS|'

You might need -maxdepth 2 instead of 1.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Sen2Agri/Sen2Agri-System/issues/28?email_source=notifications&email_token=ALRSPNLXNLCSAJWVIV4NZHLQ6DILLA5CNFSM4JLJJRQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJFVBPI#issuecomment-575361213, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALRSPNP7NTV5LA62P6Y4R6DQ6DILLANCNFSM4JLJJRQQ .

smfcrts commented 4 years ago

I've got this error

cannot rename /mnt/archive/maccs_def/morocco/l2a/S2A_MSIL2A_20190614T112121_N0207_R037_T29SMS_20190614T114731.SAFE/sedI0PNHV: Permission denied

lnicola commented 4 years ago

I'm not sure what that would happen. You ran it under the sen2agri-service system account, right? Does it not have permissions on that directory?

smfcrts commented 4 years ago

yes i ran under sen2agri-service system account and it have permission on that directory

smfcrts commented 4 years ago

@lnicola I overcome the problem, I replace in the HDR files to keep the L2A products that I already have. But when i installed Maja3.2.2 TM and i downloaded the appropriate gipp file it give's me error that gip CMN00 not found.