Provide docker based installation

carueda commented 4 years ago

@dwcaress, as we recently talked, I'm starting this ticket by just capturing various aspects mentioned in our email thread started a few months ago. We can elaborate on details as this progresses.

Status here last updated: 2020-05-27

[ ] Docker release: set up a mechanism to automatically build an MB-System Docker package with each major release from Github.

The mechanism exists and has been tested in a fork. All it remains is to enable it for this repo at Docker Hub.
[x] Base image: no particular preference for which Linux to use as base image.

We agreed on using CentOS 7 as base image.
[x] GMT: docker image will have to have a current GMT built from source just as it does MB-System.
[x] GUIs: existing visualization tools use X11/Motif/OpenGL
- [x] All of them seem to be working on CentOS7
- [x] MacOS: X11-only graphical tools are working
- [x] MacOS OpenGL (thanks Dave for the solution)
[x] Other: there needs to be a way to display postscript and images, and to edit text files.

evince, gedit, vim are included in the image.
[x] Other: Access to network shares from the container.

This only needs some rather minor customization to the image regarding proper user permissions.

carueda commented 4 years ago

Captured some notes re GUIs, including some good initial tests on Mac: https://github.com/carueda/mbsystem-docker/blob/master/notes/gui.md

Note: for now, separate repo used while putting together the major elements of this task.

EDIT(April'20): Now here: https://github.com/dwcaress/MB-System/tree/master/docker/gui

carueda commented 4 years ago

As discussed on recent email thread, just adding some comments in case of any misunderstanding with this task, in particular regarding other related activities:

The goal here is to create a single docker image that runs on whatever host platform the end user wants to use. A docker engine is basically the only requirement on the target platform. In other words, this will be one other "release" option for end users. (I can imagine some more expert users still wanting to build on native OSs as they are used to, and this task is in no way preventing that.)

For this image we can use whatever Linux base image where MBSystem is typically built. We are moving forward with Centos7 mainly because of the extra expertise we have around here at MBARI on that platform, but, again, Ubuntu or other linux should be ok.

The travis setup, on the other hand, has a different focus: build and test the system on multiple target OSs.

Of course, the docker image release build should also incorporate whatever appropriate automated testing is in place (and I still need to look into that, btw).

carueda commented 4 years ago

As part of PR #1062 , the image build now includes make check.

carueda commented 4 years ago

For next steps regarding GUI, input would be most welcome about the following:

OK to proceed with the understanding that X11 (or similar) on the host would be the mechanism to use?

Or perhaps any comments for considering possible alternatives, e.g., VNC (either directly or even via a web browser)?

dwcaress commented 4 years ago

Carlos, Yes, for now the graphical utilities are all X11/Motif or X11/Motif/OpenGL. However, Tom O'Reilly is making progress towards Qt versions of the tools. There will be a period where there are both X11 and Qt tools, and then we will migrate entirely to Qt. Generally I find it possible to run graphical tools remotely either through VNC or from a shell using ssh -X or ssh -Y.

carueda commented 4 years ago

Thanks Dave. I'm then focusing on the X11-based setup. (btw, I'm not sure how Qt would "circumvent" the need for such (or similar) setup, but I'm not really familiar with Qt at all.)

dwcaress commented 4 years ago

Yes, I think that you are correct for what you are working on - Qt on Linux is layered on top of X11, so there would be little change other than losing the Motif dependence. On Macs Qt is layered on top of Aqua, and has nothing to do with X11.

carueda commented 4 years ago

Hi @dwcaress , @cdsferreira

In my testing so far, the (dockerized) GUI programs are opening their graphical windows apparently OK on both MacOS and Linux/CentOS7.

But mbvelocitytool is failing on that Linux box as follows: (see below for how it looks on a MacBook pro)

$ export MBSYSTEM_IMAGE=mbari/mbsystem:5.7.6beta32
$ ./mbsystem.sh mbvelocitytool
...
Warning: Cannot convert string "-*-helvetica-bold-r-*-*-*-140-75-75-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-140-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-120-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-240-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-medium-r-*-*-*-140-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-180-*-*-*-*-iso8859-1" to type FontStruct

Failure to load font using XLoadQueryFont: -*-fixed-bold-r-normal-*-13-*-75-75-c-70-iso8859-1
    Source file: mbvelocity_callbacks.c
    Source line: 420
Program <MBvelocitytool> Terminated

So, while the needed font resources are available on MacOS (using XQuarts), they are not on the mentioned linux box.

The part in mbvelocity_callbacks.c where the failure and exit happens is this, where xgfont is defined here.

Any ideas? If it helps, the mentioned linux box is as follows:

$ cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
$ uname -a
Linux tethysdocker.shore.mbari.org 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Please let me know when you get a chance, thanks.

mbvelocitytool on a MacOS:

dwcaress commented 4 years ago

Carlos,

Do mbedit and mbnavedit come up ok? That would be weird because all three programs use the same font definitions. The problem is already weird because it's is failing on a call to access a generic fixed width font, which is to say any fixed width font will do. There's no way that an X11 server starts up without having at least one fixed width font loaded.

One suggested hypothetical workaround would be to also install the text editor nedit. Like the MB-System graphical tools, nedit is a Motif/X11 program. I generally install nedit on all Linux machines, but don't consider it an MB-System dependency. It might be that installing nedit will cause the installation of the missing font packages.

Cheers, Dave

carueda commented 4 years ago

Do mbedit and mbnavedit come up ok?

Oh, I thought I had tested those other programs as well on that linux box, but no, sorry. You are right, they are also failing with a similar error. I got the wrong impression because mbnavadjust seems to be working although, admittedly, not very cleanly:

One suggested hypothetical workaround would be to also install the text editor nedit.

Good idea. My "reference" programs for a sort of very basic testing on a new target have been the traditional xeyes and xclock, which are working OK (only the latter with a warning on the console about "Missing charsets in String to FonstSet conversion"). I don't have privs to install nedit on that box right now but will see what I can do. Thanks!

carueda commented 4 years ago

On the new centos7 vm just created I was able to install nedit (from sources) and run it:

I just had to yum install libXt-devel.x86_64 motif-devel.x86_64 for nedit's make linux to complete successfully.

However, still seeing the same error with the MB-System programs, which, in the mbedit case looks like:

Warning: Cannot convert string "-*-helvetica-bold-r-*-*-*-140-75-75-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-140-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-120-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-240-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-medium-r-*-*-*-140-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-180-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-courier-*-r-*-*-*-90-*-*-*-*-iso8859-1" to type FontStruct

Failure to load font using XLoadQueryFont: -*-fixed-bold-r-normal-*-13-*-75-75-c-70-iso8859-1
    Source file: mbedit_callbacks.c
    Source line: 345

Will continue looking into finding some way for the "missing font packages" to get installed.

carueda commented 4 years ago

Ok, seems like yum install xorg-x11-fonts-misc did the trick, at least to some extent (still some warnings displayed):

carueda commented 4 years ago

@dwcaress It seems I'm now be able to make the dockerized programs access the atlas share from the centos7 vm created at mbari. I'll be tuning up the script and docker image for that particular use case. In the meantime, when you get a chance, can you please indicate some tests that I can try to at least see some actual datasets being loaded in the programs? Attached is just a quick screenshot showing a file selection in one of the programs with listings under the atlas share. Thanks.

carueda commented 4 years ago

As docker_user on the centos7 box, and via VNC from my mac, a basic GUI test with mbedit accessing files under the atlas share:

carueda commented 4 years ago

Per the mbeditviz run below, noting the following:

libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast

Any ideas how to solve this @dwcaress ?

Selected one of the listed files and clicked "view selected files". After a few moments, nothing was displayed and the console shows:

...
Generating *.resf file by rerunning mbprocess:
    mbprocess -I /opt/MBSWorkDir/SeafloorMapping/swathdata/surveys/UH/Garcia/298/KMmbb022981840.d01 -P
--: Data not processed - up to date but overridden - locked: 
    Input:  /opt/MBSWorkDir/SeafloorMapping/swathdata/surveys/UH/Garcia/298/KMmbb022981840.d01
    Output: /opt/MBSWorkDir/SeafloorMapping/swathdata/surveys/UH/Garcia/298/KMmbb022981840p.mb121
    Locked by program <MBeditviz> run by <unknown> on <mbscentos7.shore.mbari.org> at <Mon Apr 27 18:09:38 2020>
loaded swathfile:/opt/MBSWorkDir/SeafloorMapping/swathdata/surveys/UH/Garcia/298/KMmbb022981840p.mb121.fbt

Grid bounds (longitude latitude): -157.7207913 -157.5943386 21.0362689 21.1325399
Grid bounds (eastings northings): 632837.143 646063.349 2326694.326 2337452.131
Altitude range: -1.010 614.030
Cell size:12.281
Grid Dimensions: 1078 877

Generating Grid:
----------------
Grid bounds (longitude latitude): -157.7207913 -157.5943386 21.0362689 21.1325399
Grid bounds (eastings northings): 632837.143 646063.349 2326694.326 2337452.131
Cell size:12.281
Grid Dimensions: 1078 877
Algorithm: Footprint
Interpolation: 0

libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast

dbg2  MBIO function <mb_proj_init> called
dbg2  Input arguments:
dbg2       verbose:    5
dbg2       projection: UTM04N

dbg2  MBIO function <mb_proj_init> completed
dbg2  Return values:
dbg2       pjptr:           0x6a92790
dbg2       error:           0
dbg2  Return status:
dbg2       status:          1
SECONDARY GRID PROJECTION:1 0x6a92790 UTM04N

dbg2  MBIO function <mbview_setcolorchangenotify> called
dbg2  MB-system Version 5.7.6beta32
dbg2  Input arguments:
dbg2       verbose:                   0
dbg2       instance:                  0
dbg2       mbview_colorchange_notify: 0x404e70

carueda commented 4 years ago

Ok, seems like the mesa-dri-drivers package, which was missing, solves the "dri" driver errors on the centos7 based image.

Here's a test running mbeditviz with input /opt/MBSWorkDir/SeafloorMapping/2019/20190613m1/datalist.mb-1:

with output on the console as follows:

bash-4.2$ mbeditviz
Warning: Cannot convert string "-*-helvetica-bold-r-*-*-*-140-75-75-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-courier-*-r-*-*-*-90-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-240-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-120-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-140-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-medium-r-*-*-*-140-*-*-*-*-iso8859-1" to type FontStruct
Warning: Cannot convert string "-*-times-bold-r-*-*-*-180-*-*-*-*-iso8859-1" to type FontStruct

dbg2  Program <MBeditviz>
dbg2  MB-system Version 5.7.6beta32
dbg2  Control Parameters:
dbg2       mbev_verbose:        0
dbg2       help:                0
dbg2       input_file_set:      0
dbg2       delete_input_file:   0
dbg2       input file:          
loaded swathfile:/opt/MBSWorkDir/SeafloorMapping/2019/20190613m1/20190613_163139p.mb88.fbt

Unable to open input file:
  /opt/MBSWorkDir/SeafloorMapping/2019/20190613m1/20190613_163833.mb88
File locked by <mappingauv> running <mbprocess>
on cpu <ross.shore.mbari.org> at <Fri Jun 14 13:20:21 2019>
loaded swathfile:/opt/MBSWorkDir/SeafloorMapping/2019/20190613m1/20190613_164529p.mb88.fbt
loaded swathfile:/opt/MBSWorkDir/SeafloorMapping/2019/20190613m1/20190613_165226p.mb88.fbt
loaded swathfile:/opt/MBSWorkDir/SeafloorMapping/2019/20190613m1/20190613_165243p.mb88.fbt
loaded swathfile:/opt/MBSWorkDir/SeafloorMapping/2019/20190613m1/20190613_165937p.mb88.fbt
loaded swathfile:/opt/MBSWorkDir/SeafloorMapping/2019/20190613m1/20190613_170633p.mb88.fbt

Grid bounds (longitude latitude): -121.9451990 -121.9371230 36.6923051 36.7007676
Grid bounds (eastings northings): 594220.066 594951.850 4061257.964 4062204.002
Altitude range: 24.949 37.721
Cell size:0.754
Grid Dimensions: 971 1255

Generating Grid:
----------------
Grid bounds (longitude latitude): -121.9451990 -121.9371230 36.6923051 36.7007676
Grid bounds (eastings northings): 594220.066 594951.850 4061257.964 4062204.002
Cell size:0.754
Grid Dimensions: 971 1255
Algorithm: Footprint
Interpolation: 0

libGL: OpenDriver: trying /usr/lib64/dri/tls/swrast_dri.so
libGL: OpenDriver: trying /usr/lib64/dri/swrast_dri.so
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /opt/MBSWorkDir/.drirc: No such file or directory.
libGL: Can't open configuration file /etc/drirc: No such file or directory.
libGL: Can't open configuration file /opt/MBSWorkDir/.drirc: No such file or directory.

dbg2  MBIO function <mb_proj_init> called
dbg2  Input arguments:
dbg2       verbose:    5
dbg2       projection: UTM10N

dbg2  MBIO function <mb_proj_init> completed
dbg2  Return values:
dbg2       pjptr:           0x18a0e120
dbg2       error:           0
dbg2  Return status:
dbg2       status:          1
SECONDARY GRID PROJECTION:1 0x18a0e120 UTM10N

dbg2  MBIO function <mbview_setcolorchangenotify> called
dbg2  MB-system Version 5.7.6beta32
dbg2  Input arguments:
dbg2       verbose:                   0
dbg2       instance:                  0
dbg2       mbview_colorchange_notify: 0x404e70

dwcaress commented 4 years ago

Carlos,

Great - that looks correct.

So yum install mesa-dri-drivers should be added to the CentOs install instructions?

Thanks, Dave

carueda commented 4 years ago

@dwcaress I'm certainly adding that instruction to the "Dockerfile" for MB-system. I would in principle say that it should probably be added to the general centos7 install instructions. However, I think this actually may depend on the actual capabilities of the target OS/machine, for example maybe there are more appropriate packages for actual nvidia drivers, or something along those lines. Make sense? (My previous experience dealing with OpenGL and drivers was a good while ago.)

carueda commented 4 years ago

@dwcaress Thanks in advance for your planned tests on a CentOS7 host.

@cdsferreira You indicated interest in testing on Ubuntu so I'm tagging you here if/when you can proceed. Please have a look at the readme under https://github.com/dwcaress/MB-System/tree/master/docker/user (also copied as the overview under https://hub.docker.com/r/mbari/mbsystem). Basically, besides docker on your target system, you will only need to have a copy of the launcher script. Thanks!

carueda commented 4 years ago

We also probably want to include some additional non-MB-System programs into the container so that they can be run from the same bash shell. These are tools that get invoked from some MB-System processing scripts. More will probably come up.
   awk, sed, vim, gedit, evince

@dwcaress awk, sed and vi (vim) are already in the container. Will try to include gedit and evince in the next version of the docker image build.

carueda commented 4 years ago

@dwcaress Just tried to build a new image with the latest 5.7.6beta36 release, but the compile phase is failing with undefined reference to 'gmt_strdup_noquote':

...
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../../src/mbio -I../../src/mbio -I/usr/include/gmt -I/usr/include/gdal -I/usr/include -g -O2 -MT mb_readwritegrd.lo -MD -MP -MF .deps/mb_readwritegrd.Tpo -c mb_readwritegrd.c  -fPIC -DPIC -o .libs/mb_readwritegrd.o
mb_readwritegrd.c: In function 'mb_write_gmt_grd':
mb_readwritegrd.c:476:5: warning: implicit declaration of function 'gmt_strdup_noquote' [-Wimplicit-function-declaration]
     header->ProjRefPROJ4 = gmt_strdup_noquote(pszResult); // allocated within GMT because it will be freed within GMT
     ^
mb_readwritegrd.c:476:26: warning: assignment makes pointer from integer without a cast [enabled by default]
     header->ProjRefPROJ4 = gmt_strdup_noquote(pszResult); // allocated within GMT because it will be freed within GMT
                          ^
...
...
/bin/sh ../../libtool  --tag=CXX   --mode=link g++ -std=gnu++11  -g -O2  -m64 -o mbauvloglist mbauvloglist.o ../../src/mbaux/libmbaux.la ../../src/mbio/libmbio.la -L/usr/lib64 -lgmt -lnetcdf   -lproj -lm
libtool: link: g++ -std=gnu++11 -g -O2 -m64 -o .libs/mbauvloglist mbauvloglist.o  ../../src/mbaux/.libs/libmbaux.so -L/usr/lib64 /opt/MB-System/src/mbio/.libs/libmbio.so -lgdal ../../src/mbio/.libs/libmbio.so /opt/MB-System/src/bsio/.libs/libmbbsio.so /opt/MB-System/src/surf/.libs/libmbsapi.so /opt/MB-System/src/gsf/.libs/libmbgsf.so -lgmt -lnetcdf -lproj -lm -Wl,-rpath -Wl,/usr/local/lib
../../src/mbaux/.libs/libmbaux.so: undefined reference to `gmt_strdup_noquote'
collect2: error: ld returned 1 exit status
make[3]: Leaving directory `/opt/MB-System/src/utilities'
make[3]: *** [mbauvloglist] Error 1
make[2]: *** [all] Error 2
make[2]: Leaving directory `/opt/MB-System/src/utilities'
make[1]: Leaving directory `/opt/MB-System/src'
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1
The command '/bin/sh -c cd /opt/MB-System/         &&      ./configure                &&      make                       &&      make check                 &&      make install' returned a non-zero code: 2

Any ideas?

carueda commented 4 years ago

I'll try the following as I just note there's a newer zberkowitz/mbsystem-deps:centos-7 image. The one on my system is 2 months old:

$ docker images | grep zberkowitz/mbsystem-deps
zberkowitz/mbsystem-deps         centos-7              43c3e7f06806        2 months ago        1.65GB

so, maybe the newer one has a more recent GMT.

carueda commented 4 years ago

No luck, same compile/link errors.

I checked and seems like 6.0.0 is still the latest GMT (tagged) release.

Looking at GMT code, I see that gmt_strdup_noquote is declared in gmt_common_string.h, which is included in gmt_dev.h, which in turn is included in mb_readwritegrd.c. So, not sure why the compiler is still complaining about

mb_readwritegrd.c: In function 'mb_write_gmt_grd':
mb_readwritegrd.c:476:5: warning: implicit declaration of function 'gmt_strdup_noquote' [-Wimplicit-function-declaration]
     header->ProjRefPROJ4 = gmt_strdup_noquote(pszResult); // allocated within GMT because it will be freed within GMT
     ^

Will try to incorporate what's included in the base image to have a bit of better control about what's actually being installed, etc.

carueda commented 4 years ago

OK, I see now. gmt_strdup_noquote was added rather recently, four months ago, https://github.com/GenericMappingTools/gmt/pull/2665/files, while tag 6.0.0 is from a year ago, or at least this file in that tag is from June'19: https://github.com/GenericMappingTools/gmt/blob/6.0.0/src/common_string.h

carueda commented 4 years ago

Ok, that mb_write_gmt_grd error is gone if using master as GMT_SOURCE_TAG for the GMT build.

Now getting:

libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../../src/mbio -I../../src/mbio -I/usr/local/include/gmt -I/usr/include/gdal -I/usr/include -g -O2 -MT mb_readwritegrd.lo -MD -MP -MF .deps/mb_readwritegrd.Tpo -c mb_readwritegrd.c  -fPIC -DPIC -o .libs/mb_readwritegrd.o
mb_readwritegrd.c: In function 'mb_write_gmt_grd':
mb_readwritegrd.c:467:5: error: unknown type name 'OGRErr'
     OGRErr eErr = OGRERR_NONE;
     ^

The GDAL/OGR include files are actually under /usr/include/gdal/. But I'm not actually seeing any #include of gdal*.h or ogr*.h files in the MB-System sources, weird.

dwcaress commented 4 years ago

The GDAL header gdal.h is included in gmt_gdalread.h which is included in gmt_dev.h, but only if GDAL is installed already when cmake is run as part of installing GMT. If GDAL isn't installed, GMT will still build, but without the GDAL related functionality, and without gdal.h being included through gmt_dev.h.

dwcaress commented 4 years ago

I thought the MB-System configure script would fail if GDAL is not installed. Perhaps GDAL is installed by the time MB-System is being built, but wasn't installed when GMT was built.

dwcaress commented 4 years ago

Got it - I think. The GDAL related headers are only included by gmt_dev.h if the conditional HAVE_GDAL is defined in the build system. I've duplicated this conditional in the MB-System build system, so the full headers are included. See #1097

carueda commented 4 years ago

Thanks @dwcaress , the build now completes and I can run the basic tests.

(I'm also publishing the resulting image mbari/mbsystem:5.7.6beta37, but my upload connection is slow so it may take some time to get reflected there.)

Note:

I had to explicitly set LD_LIBRARY_PATH as the programs were failing to load libgmt at runtime. I guess this should be fixed as part of the software build itself, but I added this to move on.
Used 5.7.6beta37 as the tag for the image per latest entry in ChangeLog, but programs are still reporting 5.7.6beta36.
Updated docker/README, take a look.

dwcaress commented 4 years ago

I've solved OpenGL display on Macs running the MB-System Docker container. This is also a fix for running MB-System OpenGL programs in an ssh session on a CentOs machine with local X11 display (e.g. ssh -X user@address or ssh -Y user@address).

A preferences default setting for XQuartz fixes this problem.

1) Quit XQuartz and shut down the Docker container.

2) In a terminal do the command: defaults write org.macosforge.xquartz.X11 enable_iglx -bool true

3) Try again. Programs like mbgrdviz and mbeditviz should now work, albeit still showing error messages and warnings.

I found this solution at: https://mu2e.fnal.gov/public/hep/computing_retired/swrast.shtml https://unix.stackexchange.com/questions/429760/opengl-rendering-with-x11-forwarding

carueda commented 4 years ago

cool, that just worked right away for me!

carueda commented 3 years ago

Just to note that I resync'ed the fork at https://github.com/mbari-org/MB-System, which triggered (as expected) the automated image build at Docker Hub. I then exercised it on a macOs essentially recreating the May 27, 2020 test above.

2021-02-16_14-07-34

MBARIMike commented 2 years ago

I'm happy to see that a Docker image is available for MB-System!

I have a need to execute MB-System commands from another Docker container and see that a suggested way to do this is via sshd. Does this seem like a reasonable capability to add? If so then I'll work on a PR.

MBARIMike commented 2 years ago

Just a note here in case anyone else runs into the same issue. The updated Dockerfile allows ssh to the mbsystem container as user root just fine. This works for my Django development system where my django container also uses root. The production system, however, does not use root. For this I add a 'docker_user' user (also used by the production django container) to the base mb-system image, e.g.: https://github.com/mbari-org/SeafloorMappingDB/blob/main/smdb/compose/production/mb-system/Dockerfile

MBARIMike commented 2 years ago

I'm having strange trouble now with trying to build the image. It fails with configure: error: Did not find PROJ library pkg-config package...:

#17 6.399 Found PROJ library libproj with PROJ4 api available
#17 6.399 configure: error: Did not find PROJ library pkg-config package...
------
executor failed running [/bin/sh -c cd /opt/MB-System/                               && ./configure --with-gdal-config=/usr/bin/         && make                                             && make check                                       && make install]: exit code: 1

What's strange is that if I use an interactive terminal session I can execute cd /opt/MB-System/ && ./configure --with-gdal-config=/usr/bin then it builds:

➜  MB-System git:(master) ✗ docker run --rm -it "mbari/mbsystem"  /bin/bash
[root@113beb26d091 MBSWorkDir]# cd /opt/MB-System/
[root@113beb26d091 MB-System]# ./configure --with-gdal-config=/usr/bin/

------------------------------------------------------------------------------
    The MB-system:    configure    12/7/2011

...

------------------------------------------------------------------------------
Installation Locations:
    executables:              ${exec_prefix}/bin
    libraries:                ${exec_prefix}/lib
    header files:             ${prefix}/include
    data files:               ${datarootdir}/mbsystem
    man pages:                ${datarootdir}/man
    Html documentation:       ${datarootdir}/doc/${PACKAGE_TARNAME}/html
    Postscript documentation: ${datarootdir}/doc/${PACKAGE_TARNAME}/ps
------------------------------------------------------------------------------
See config.log for details of configure results
See INSTALL for further build instructions and hints
------------------------------------------------------------------------------
[root@113beb26d091 MB-System]#

I've tried setting environment variables in docker/Dockerfile before the ./configure command to try and replicate the interactive terminal session, but nothing has worked. I think that something is different during the docker build than during the terminal session but I can't figure it out, but there could be something else wrong. @carueda do you have any ideas?

carueda commented 2 years ago

Hi @MBARIMike ,

I haven't looked into the MBSystem dockerization for quite some time, in fact, I'm not up-to-date as to the status of the automated builds(*). It seems they are now generated from dwcaress/MB-System (which is good as planned) instead of the fork under mbari-org initially setup while putting together the dockerization effort. (Maybe Dave could share some general comments about the status of the automated docker builds ;)

But I can provide some quick reactions (though very likely you've already thought about these ;):

configure: error: Did not find PROJ library pkg-config package...:

Do you have some log output indicating the status of the PROJ build, in particular perhaps indicating the need to update PATH or LD_LIBRARY_PATH or something along those lines?

What's strange is that if I use an interactive terminal session I can execute cd /opt/MB-System/ && ./configure --with-gdal-config=/usr/bin then it builds

Hmm, it could be that some environment is not taking (or is not set to take) effect as needed by the time of starting teh gdal build during the image build; but such env is then in effect by the time you do the gdal build afterwards?

(*) BTW, seems like the automated builds have been failing since a while ago:

MBARIMike commented 2 years ago

Looking into the proj build I see that PROJ_SOURCE_TAG wasn't being set in docker/Dockerfile:

-# So, trying master just as an interim
+# So, trying master just as an interim - should set to specific tags
 ARG GMT_SOURCE_TAG=master

-ARG PROJ_SOURCE_TAG
+ARG PROJ_SOURCE_TAG=master

 # TODO in general, confirm appropriate install mechanisms for PROJ and GMT on centos
 # (per .travis.yml, seems like the above tags are only defined for ubuntu.
@@ -36,7 +36,7 @@ COPY docker/centos/scripts/install-gmt.sh .
 RUN GMT_SOURCE_TAG=${GMT_SOURCE_TAG} ./install-gmt.sh

 COPY docker/centos/scripts/install-proj.sh .
-RUN GMT_SOURCE_TAG=${PROJ_SOURCE_TAG} ./install-proj.sh
+RUN PROJ_SOURCE_TAG=${PROJ_SOURCE_TAG} ./install-proj.sh

With this change the build now fails with:

#11 34.64   sqlite3 >= 3.11 required!

I'll work on getting this to build with ARG PROJ_SOURCE_TAG=master, but then I suggest freezing the tags to have more determinism with the builds.

MBARIMike commented 2 years ago

I pushed a few changes that fixes the Docker build on my system. I wonder @dwcaress if the automated build will work any better now. (Note that https://github.com/dwcaress/MB-System/pull/1259 also includes the commit that's in another PR from me.)

dwcaress commented 2 years ago

I've merged these changes with master and with my development branch caress-tmp.

carueda commented 2 years ago

Just adding a quick update here regarding the recent setup and satisfactory testing on a Windows 11 machine:

https://github.com/dwcaress/MB-System/blob/master/docker/user/README-win11.md

referenced from

https://github.com/dwcaress/MB-System/blob/master/docker/user/README.md

carueda commented 12 months ago

Captured an exercise today trying to successfully run a graphical program on a macOS Ventura (intel):

https://gist.github.com/carueda/e4e44aabc5db2bd14a41f771d15faa11

MBARIMike commented 11 months ago

Has the automated at Docker Hub stoped working?

I need to use the mbgrd2gltf tool that was added on 7 Jun 2023. The release 5.7.9beta59 from 2 weeks ago should include this tool, but I don't see it in one of the builds in Docker Hub. The latest tag there is 5.7.6beta37

The mechanism as mentioned in https://github.com/dwcaress/MB-System/issues/807#issuecomment-780150760 seems to apply to a fork. Does this mechanism need to be enabled for upstream/master?

dwcaress / MB-System

Provide docker based installation #807