New auxiliary data warning.

[sharkinsspatial]

The most recent patched release of Fmask_4_4 includes this commit commenting about the referencing of auxiliary data. We (and I believe the majority of people) using the standalone executable are doing so through the run_Fmask_4_4.sh shell entrypoint which is included with the installer (but not shown in the source Matlab repository here). Is it possible for you to modify this as part of the installer package so that it includes the ${exe_dir}/AuxiData reference in the Fmask_4_4 call.

Additionally, I believe the majority of the community has been running Fmask without referencing the auxiliary data (as this has been silently failing with no notification). Can you describe the implications of the resulting output being created without the auxiliary data?

I'm cc'ing @j08lue here as well as I know they are standalone users via https://github.com/DHI-GRAS/fmaskilicious.

[qsly09]

@sharkinsspatial Many thanks for your insightful comments, and sorry for any inconvenience. In the updated Fmask 4.4, the path of the auxiliary dataset is one of the inputs when calling the Fmask, which can be set by the command like <Fmask_4_4 'the path'>.

If we did not take the auxiliary dataset successfully included, the Fmask 4 will get back to the major routines of Fmask version 3.3 (USGS Landsat QA band is being generated by using this version 3.3), but this may include more commission errors from clear-sky pixels, especially for mountainous and water areas. It can be also noted that the Fmask 4 without the auxiliary data can still significantly improve cloud and cloud shadow detection, particularly for Sentinel-2 data, because of new features nothing about the DEM and water layer.

Thanks to all of you for helping us to improve the Fmask tool further.

[sharkinsspatial]

@qsly09 Are you able to update the run_Fmask_4_4.sh script packaged with the installer as I suggested?

[qsly09]

@sharkinsspatial I have no idea about the run_Fmask_4_4.sh. Is this the one you created on your side? Could you please share the run_Fmask_4_4.sh ?

[sharkinsspatial]

@qsly09 It is packaged inline with the application (I'm not sure what steps you take to create the mcr install) but it is included in the application directory next to the Fmask_4_4 executable and the AuxiData directory. It's contents are

#!/bin/sh
# script for execution of deployed applications
#
# Sets up the MATLAB Runtime environment for the current $ARCH and executes
# the specified command.
#
exe_name=$0
exe_dir=`dirname "$0"`
echo "------------------------------------------"
if [ "x$1" = "x" ]; then
  echo Usage:
  echo    $0 \<deployedMCRroot\> args
else
  echo Setting up environment variables
  MCRROOT="$1"
  echo ---
  LD_LIBRARY_PATH=.:${MCRROOT}/runtime/glnxa64 ;
  LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/bin/glnxa64 ;
  LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/sys/os/glnxa64;
  LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/sys/opengl/lib/glnxa64;
  export LD_LIBRARY_PATH;
  echo LD_LIBRARY_PATH is ${LD_LIBRARY_PATH};
# Preload glibc_shim in case of RHEL7 variants
  test -e /usr/bin/ldd &&  ldd --version |  grep -q "(GNU libc) 2\.17"  \
            && export LD_PRELOAD="${MCRROOT}/bin/glnxa64/glibc-2.17_shim.so"
  shift 1
  args=
  while [ $# -gt 0 ]; do
      token=$1
      args="${args} \"${token}\""
      shift
  done
  eval "\"${exe_dir}/Fmask_4_4\"" $args
fi
exit

[qsly09]

Could you try to change the call commands to the one like "\"${exe_dir}/Fmask_4_4 the directory path of \\""?, which may force to set up the path.

[sharkinsspatial]

Modifying

eval "\"${exe_dir}/Fmask_4_4\"" $args

to

 eval "\"${exe_dir}/Fmask_4_4\"" ${exe_dir}/AuxiData $args

will successfully locate the AuxiData when running the command via

/usr/local/MATLAB/application/run_Fmask_4_4.sh /usr/local/MATLAB/v910

Is there any reason a user would want to run the standalone Fmask without using the auxiliary data? If not, I would suggest that you modify run_Fmask_4_4.sh to use this AuxiData argument and include it the installer. The auxiliary data is packaged with the installer, so in my thinking there is no reason that shell executor should not make use of it automatically.

[qsly09]

Great! The Linux installer cannot automatically find the dataset because it is hard for the Matlab code to extract the exe's directory on Linux. Anyway, let's move to this manual setting for Linux standalone, which would be safe. I will set a link to jump to this issue page for any users who may implement the Linux standalone. Thanks again.

[sharkinsspatial]

@qsly09 Can you clarify what you mean by "manual setting" here? Will you be providing an updated version of run_Fmask_4_4.sh in the installer or not?

[qsly09]

@sharkinsspatial 'manual setting' here means we need to set the directory of the auxiliary data when we call the Fmask standalone, as you did here. Sorry for any confusion. BTW, we have further updated to version 4.5 now, not too much modification, which just applies a static seed random generator. Then, I have attached the new run_Fmask_4_5.sh (with your modification) in the install package, but actually, the installer was automatically generated by MATLAB compiler, including the shell script run_Fmask_4_5.sh in the application directory after installing, and I can't edit it when building the standalone. Thus, I just packaged a text file to explain this, and also provided the modified run_Fmask_4_5.sh within. Hope this can make it more easily implemented. Thank you very much.

[sharkinsspatial]

Closed by https://github.com/NASA-IMPACT/hls-base/pull/26

[demaet]

Beware: if you are using any other (numeric) arg it will fail.

The order is 1) numeric args, 2) path to AuxiData.

The correct line is then:

eval "\"${exe_dir}/Fmask_4_5\"" $args ${exe_dir}/AuxiData

Thanks for the code, by the way !