Questions on using python fmask on user produced Sentinel TOA reflectances (particularly ImageAngle)

[BJKUNS]

Hi! Thanks for making python fmask! I usually don’t work in a python environment, but I found the installation straightforward on my ubuntu laptop, and it works great.

This is either a question or a feature request. I’ve read on the python fmask website that most people use the --safedir option to access and apply the fmask to Sentinel images, and that the older way which necessitated, among other things, the separate creation of Anglefiles is now obsolete.

I would like to use my own stack of Sentinel TOA reflectance files in python fmask, instead of using the -- safedir option. I want to run my own crop mask on the Sentinel files first to see if that will reduce the errors of commission. One question I have is will python fmask work with other file types as input files, i.e. geotiffs? (I assume it does, but just wanted to check)

Is there any documentation on how one would go about using python fmask on a stack of TOA reflectances produced by the user. In particular, I wonder if there is some documentation on what one would need to do to to produce your own Anglefiles. I usually work in an R environment and my python abilities are not so good.

The feature request part of it is if there is a way to produce the ImageAngle files from the command line. That might be too much work for you, particularly if there is not so much interest in such a feature, but I thought I would throw that out there.

Thanks again for making python fmask!

[neilflood]

Thanks for your interest.

There are several questions there.

1. re: other input file formats. Buried underneath all this code, we are using GDAL to read and write the image files. This means that it will read most of the usual image formats, definitely including GeoTIFF.
2. re: producing angles image files from the command line. There is a command line wrapper called fmask_sentinel2makeAnglesImage.py included in the package. This takes the granule-level XML file (normally called MTD_TL.xml) as input, and writes an image file as output. The angles image file is at 5km pixel size, as this is how the data is supplied by ESA.
3. If you want to go further than this, and create your own main script which handles the files in a different layout, this is quite OK and within the intended design, but you will need to be OK with writing Python. There is no specific documentation for this, but I suggest that you start by reading the code in our main module sentinel2Stacked.py, and understand how that sets up the different files. This is where it knows about the layout of the ESA SAFE directory, so you would be working out the equivalent for how you wish to organise the files.
4. I will just add that the reason for the separation of the cmdline modules from the actual main command line scripts is to make is easier to work on Windows, which has weird mechanisms for finding the Python interpreter and so on. Since you are on Linux, you can largely ignore that if you wish.

[BJKUNS]

Thanks for your answer! I got that command line wrapper for AngleImage to work fine. That was straightforward. I then tried to use my own stack (.tif) with the AngleImage I produced to make a cloud mask. It worked in that it produced a cloud mask, but it took a suspiciously short amount of time. Also, I ran fmask on the same scenes using the --safedir default option (on the original files), and the cloud mask produced there is better. Shouldn't I get the same result? They are the same files, one saved as a stack (.tif), while the others are in the original file structure. I will tinker with it during the week to see if there is something I have done when I produced my own stack, but I wonder if you have any words of advice on how the user-produced stack should be created to work effectively.

My command looked like this:

fmask_sentinel2Stacked.py -o cloud_mask.img -z angle.img -a mystack.tif

Another question. If I run a crop mask on the original Sentinel files, and save them as .tif files, but otherwise with the same files names and in the same files structure as the original files, will the safedir option work?

[neilflood]

Yes, you should get exactly the same result. My first guess is that the mystack.tif contains the layers in the wrong order. The correct order should be in order of band number, with band 8A just after band 08.

re: crop mask files mixed into the safe dir. Well, I cannot guess exactly what this would look like, but the --safedir option searches explicitly for .jp2 files, so if you have called your files .tif, then it probably should work. You will have to test it to be sure, though.

[BJKUNS]

Thanks again for the answer. Yes, the mystack.tif did contain the layers in the wrong order, with 8A ending up in the wrong place. Now that I have the layers in the right order in the stack, I get almost the same result, but there are still some minor differences. However, I'm pretty sure that this has to do with the creation of the mystack.tif file. For example, the .tif layers are smaller files compared to the corresponding SAFE layers. I created the layer stack in QGIS using standard defaults but retaining the unsigned 16 bit integer data format. Anyway, that's an issue on my end, and I'll play around with it to see what's going on. Also, I can't do it this week, but I'll test the option of inserting cropped .tif files into the safe dir format, and report back if that works. Also, I think I could use the reticulate package in R to call the python functions in fmask. So I might try out that route as well. Again, thanks for your help!

[neilflood]

OK, thanks. I will close this issue now, as that seems to have solved all your problems. Good luck.