Installation and usage notes

[jonas-eberle]

Hey Volly,

wow, I am really impressed by the shiny app. That is the way forward! Congratulations.

1) Update Readme

Please update your readme, the path to the installer script is not correct: wget https://raw.githubusercontent.com/openforis/opensarkit/master/bins/Install_OFST/installer_ubuntu1604.sh

The correct name of the install folder is Install_OST.

2) Update installation script

I tried to use the shiny app for Sentinel-1. Using the data inventory step it resulted in the app should maybe report if there are no results.

In my case there were some commands missing, you can easily add them to your installation script:

curl
spatialite (installed via spatialite-bin)

3) Data download

I only get the following messages (repeated) in the log file: Downloading Product 1 of 71 (Granule: S1A_IW_GRDH_1SDV_20170611T170710_20170611T170735_016989_01C495_C701)... failed, having another try.

Is there a limit for download tries? If not, you should integrate one. Also, I cannot find a way to cancel the download process beside stopping the shiny app.

I found the reason of the download failures: I used a wrong password due to the english keyboard ;-) Now, the downloads work fine. Maybe the wrong authentication can be catched somewhere and reported to the user?

4) GRD to RTC processor

Will report later after the processing has finished...

5) Some general notes

• It would be good to show some log file entries in the shiny app so that users see the progress.
• There should be a link to the local log file somewhere in the app
• Do you plan to publish a Dockerfile for the use with Docker? I think this is not a big effort as you already have your installation script automated.

I also really appreciate the documentation beside the individual steps.

Cheers from Jena University, Jonas

[BuddyVolly]

Hi Jonas,

thanks for your elaborated feedback and yes, there are still tons of things to take care of, especially reflecting error handling.

1 & 2, I will do immediately.

3. No there is no limit to the number of attempts, and yes it should have a message stating at least for authorization failures. Bridging the world of shiny and bash can sometimes be a hassle. Aborting processes should however be included as well.

4. Yes, that can take a while...especially for lots of scenes

5) link to log file entry --> really nice I idea, with shiny it is always a bit tricky getting the progress. That could be a nice solution, especially for the long lasting tasks.

Docker file, yes we will do this very soon.

Documentation, we will do demos, that basically guide you through the functionalities and let you learn SAR also very soon. A more detailed documentation will follow.

Thanks Jonas, and let me know if you have other issues or wishes! As you can imagine, time is the only limit and I am always happy about feedback and suggestions.

Cheers, Volly

[BuddyVolly]

P.S. Number 3 is also solved and will be released with the next push. An error will occur if there are authorization problems. And it will try max of 50 times for one scene, otherwise it is going to exit. The latter might change, since we have to keep in mind bad connections in countries where internet connection is unstable. On the other hand we do not want a script to run endlessly.

[BuddyVolly]

P.P.S. Your wishes of point 5 will come true soon. I am working on it, but have to do some fundamental changes in the shiny app that will take some days. There will be a real-time output of the logfile, and also a button for cancelling the processing. Once this is done, a dockerfile will be created and I will work on the demos/documentantation.

[jonas-eberle]

Great. I really looking forward for the dockerfile, then I can install it on my server rather than on a local virtualbox. I made the experience that I cannot use the shiny app from another tab while a process is running. Is that really the case?

[BuddyVolly]

Hi Jonas,

you gave me a hard week, but, after all, all of those issues mentioned are basically solved, though not implemented yet. This includes an abort processing button, as well as a real-time output of the ongoing processing steps. Error handling will be really basic, but it will be improved step-by-step. At least authorization failures are going to be handled, and successfulness of processes are checked.

Asynchronous processing in R and shiny, respectively, is actually not really possible. We could do it, because the only thing we are doing from shiny is calling bash scripts and so we could detach them from the Rsession. We then use kind of callback functionality in R shiny. This should theoretically also allow for parallel use of different tabs (need to test this though when it is properly implemented). As an alternative, you should be able to start different sessions in parallel? Otherwise you could use the command line. All commands start with "ost*" , apart from the ALOS K&C ones, since they use gnu parallel and start with "post*".Usage is shown by running the script without arguments.

I will prepare a docker image once the changes have been implemented. We also want to have a stable tagged version on github soon. This could possibly be the first one. But both will take another 1-2 weeks. So please be patient! In the meantime, do not hesitate to suggest more improvements. There is still tons of stuff to do, but it is good to prioritize things based on user demands/experience.

[jonas-eberle]

Hi Volly,

I hope the hard work was nevertheless worthwhile ;-) and I am looking forward for this improvements. There is currently no need to hurry.

Regarding the parallel use of the app, I thought about install it on a server and others can use it (the current setup would not allow this). But maybe there is no need to change your app, rather than changing the setup environment running this app. I think there could be a multi-user environment using a docker instance for each user? Then you do not need to integrate parallel use an the shiny app. I have just discovered Shinyproxy:

ShinyProxy is your favourite way to deploy Shiny apps in an enterprise context. It has built-in functionality for LDAP authentication and authorization, makes securing Shiny traffic (over TLS) a breeze and has no limits on concurrent usage of a Shiny app.

It seems that you only need a docker file for this. I can try this out once you have created the docker file.

[BuddyVolly]

Hi Jonas,

so we are using this on Sepal, where the whole architecture is somewhat different since there it is "just" one module out of a couple. We do not have experience with the Shinyproxy, but from our understanding here this could be an easy way for you. Though we cannot assist you with the details of ShinyProxy, we will use the dockerfile template of shinyproxy in order to assure you compatibility (cross fingers).

One feature that is added as well is to make use of the RAM for temporary files. If you run the stuff on a server with enough ram, you should consider this. OST will check if there is a folder called "/ram" mounted. In case this is big enough for the demand in the TMP folder, it will use it automatically. First tests showed a considerable reduction in processing time, i.e. about 33% of what it was before using SSDs.

I'll come back to you once the dockerfile is created. Btw...do you have experience using docker containers on Windows?

Cheers, Andreas

[jonas-eberle]

Thank you for the updates. I will try it with ShinyProxy once the dockerfile is ready.

I tried to get docker running on Windows (7)... but this seems to be a bit special. Maybe it is getting better by using Windows 10, then you can use Docker for Windows instead of the Docker Toolbox.

[BuddyVolly]

Hi Jonas, I am happy to inform you that there is a docker image available. Although having done lots of fundamental changes to OST, I did not have the time to test everything. So problems are expected, although I could produce ALOS and Sentinel-1 data sucessfully.

Regarding docker: We used kind of the template docker file from shinyproxy, which should allow you for deploying OST on a server. Once you have docker installed, run: docker run -it --name ost -v /home/user/:/root -p 3838:3838 openforis/opensartoolkit

• v is the mount flag, since the disk space in your docker container won't be sufficient. Change this to your needs
• p are the ports. You can change this if you want to have it run on another port, but inside docker it will be 3838

(Un)fortunately we do not have Windows 10 here on our pcs ;) If you have the chance to try it out you would make me really happy. And of course, any feedback is most appreciated.

Cheers, BV

[jonas-eberle]

Thank you so much, Andreas. I have installed the docker image and currently it runs without shinyproxy on my linux host system. Using Shinyproxy will then be the next step.

Unfortunately I cannot help with Windows 10 at the moment. But having a docker image ready to use should also work on Windows (in general...).

Interesting: While downloading S1 data using the docker image, I can open another browser window and use the same docker instance at the same time.