USACE / go-consequences

A lightweight consequences computational engine written in Go
MIT License
14 stars 11 forks source link

Strange libgdal.so.28 issue #120

Open jlgutenson opened 3 weeks ago

jlgutenson commented 3 weeks ago

Describe the bug I'm having an issue with a libgdal.so.28 dependency somewhere in go-consequences when setting up and running go-consequences using Docker Desktop on Windows. The error is:

/app/main: error while loading shared libraries: libgdal.so.28: cannot open shared object file: No such file or directory

after attempting to run a flood consequence assessment with the command:

docker run --mount type=bind,source="C:\Users\jlgut\OneDrive\Desktop\go-consequences",target=/data go-consequences-jlg:latest ./main "/data/joseph_test.json"

To Reproduce Steps to reproduce the behavior:

  1. Go to a local directory of the go-consequences Github repo (e.g., C:\Users\jlgut\OneDrive\Desktop\go-consequences).
  2. Modify Dockerfile.prod because of initial issue with not finding the /main executable:
    
    # Stage 1: Builder
    FROM debian:bullseye AS builder

ENV TZ=America/New_York ENV PATH=/go/bin:$PATH ENV GOROOT=/go ENV GOPATH=/src/go

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone &&\ mkdir -p /go &&\ mkdir -p /src/go &&\ apt update &&\ apt -y install build-essential &&\ apt -y install gdal-bin gdal-data libgdal-dev &&\ apt -y install wget &&\ wget https://golang.org/dl/go1.19.5.linux-amd64.tar.gz -P / &&\ tar -xvzf /go1.19.5.linux-amd64.tar.gz -C / &&\ apt -y install vim &&\ apt -y install git

WORKDIR /app RUN git clone https://github.com/USACE/go-consequences.git WORKDIR /app/go-consequences RUN go mod download RUN go mod tidy RUN go build -o main .

Stage 2: Production

FROM ghcr.io/osgeo/gdal:ubuntu-full-3.8.3 AS prod

Install GDAL and related libraries

RUN apt update && apt -y install gdal-bin gdal-data libgdal-dev

WORKDIR /app COPY --from=builder /app/go-consequences/main ./

Ensure the GDAL library is available

RUN ldconfig

ENTRYPOINT ["/app/main"]

4. Build the Docker Image with the command ```docker build -f Dockerfile.prod -t go-consequences-jlg .```
5. Attempt to run go-consequences from command line ```docker run --mount type=bind,source="C:\Users\jlgut\OneDrive\Desktop\go-consequences",target=/data go-consequences-jlg:latest ./main "/data/joseph_test.json"```
6. See error: ```/app/main: error while loading shared libraries: libgdal.so.28: cannot open shared object file: No such file or directory```

**Expected behavior**
I was expecting to have go-consequences run this test-case (i.e., joseph_test.json):

{ "structure_provider_info": { "structure_provider_type": "NSIAPI" }, "hazard_provider_info": { "hazards": [ { "hazard_parameter_type": "depth", "hazard_provider_file_path": "/data/MO_FABDEM_ARC_Flood_Forecast_20241030.shp" } ] }, "results_writer_info": { "results_writer_type": "JSON", "output_file_path": "/data/MO_FABDEM_ARC_Flood_Forecast_20241030.json" } }

HenryGeorgist commented 3 weeks ago

@jlgutenson cool work. looks great. do you know that I publish a go-consequences container that you can call like you want?

I also notice that you build on the gdal image, and that you do not grab build essential. In my prod container i have the following lines:

RUN apt update &&\
    apt -y install build-essential &&\
    apt -y install pkg-config &&\
    apt -y install gdal-bin gdal-data libgdal-dev

I can understand not wanting build essential in the prod container, but that is a potential for where this might differ.

I see that you have your hazard provider as a shapefile - did you update the code locally to work with that hazard provider type?

jlgutenson commented 3 weeks ago

Hey @HenryGeorgist, good to hear from you! Thanks for the quick response!

I didn't know you had your image published. That sounds like exactly what I need. Is it this guy?

That shapefile was actually supposed to be tiff file/depth grid reference. Good eye!

If so, I tried running docker run --mount type=bind,source="C:\Users\jlgut\OneDrive\Desktop\go-consequences",target=/data williamlehman/go-consequences:v0.0.2 ./main "/data/joseph_test.json" which is a modification to your command here.

However, I'm not finding the /main executable when I run your published image. I get this error: docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/main": stat /main: no such file or directory: unknown.

HenryGeorgist commented 3 weeks ago

hmm. that looks ok. did you try v0.0.1?

HenryGeorgist commented 3 weeks ago
C:\Modeling>docker run -v C:\Modeling:/data williamlehman/go-consequences:v0.0.1 ./main "/data/config.json"
Connecting to: /data/depthgrid.tif
Getting bbox
-92.537687,29.911019,-92.333679,29.911019,-92.333679,29.823433,-92.537687,29.823433,-92.537687,29.911019
https://nsi.sec.usace.army.mil/nsiapi/structures?bbox=-92.537687,29.911019,-92.333679,29.911019,-92.333679,29.823433,-92.537687,29.823433,-92.537687,29.911019&fmt=fs

C:\Modeling>docker run -v C:\Modeling:/data williamlehman/go-consequences:v0.0.2 ./main "/data/config.json"
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/main": stat /main: no such file or directory: unknown.

i can reproduce it on version 0.0.2, looks like the exported command was /main instead of ./main let me try running interactively

HenryGeorgist commented 3 weeks ago
C:\Modeling>docker run -v C:\Modeling:/data -it --entrypoint /bin/bash williamlehman/go-consequences:v0.0.2
root@1804b072b3db:/app# ls
main
root@1804b072b3db:/app# ./main
./main: error while loading shared libraries: libgdal.so.28: cannot open shared object file: No such file or directory

reproduced with my checked in prod container.

jlgutenson commented 3 weeks ago

hmm. that looks ok. did you try v0.0.1?

Good suggestion!

I can confirm that your v0.0.1 image is running. But my outputs (I tried json, geopackage, and shapefile) with v0.0.1 are not complete once the analysis ends. That might be a connection issue on my end.

jlgutenson commented 2 weeks ago

hmm. that looks ok. did you try v0.0.1?

Good suggestion!

I can confirm that your v0.0.1 image is running. But my outputs (I tried json, geopackage, and shapefile) with v0.0.1 are not complete once the analysis ends. That might be a connection issue on my end.

Hey @HenryGeorgist, after further analysis, I'm having some issues with the v0.0.1 image's outputs that I don't believe are memory or internet connection related. I had ChatGPT analyze the file and it wasn't able to clean the JSON to make it usable in a GIS. Here is an attached JSON output for reference. depth_max_clipped2.json

HenryGeorgist commented 2 weeks ago

on first glance it is missing a closing square brace at the end ] of the features array, and then it would need to close the feature collection object with a curly brace } In json specification array items are separated by commas except for the last item in the array, so there is a comma at the end of your file that also needs to be deleted.

the end should look essentially like this }}]} it currently looks like this }},

this should happen in the "Close" methodology on the results writer. it is possible that the close method wasnt defered in version 0.0.1 for the geojson results writer.

i typically use geopackage. but if the close method is not deferred on it too then we would have a problem of not flushing the last transaction. i think geojson and editing the last two characters may be enough....

chat gpt? seriously?

jlgutenson commented 2 weeks ago

I gave }}]} at the end a whirl. No luck, unfortunately.

With v0.0.1, the Geopackage option keeps giving me only one damaged building. The one building is in the depth grid and pulling the correct depth but only writing the one location for some reason.

I tried the shapefile option as well and it was completely empty.

With ChatGPT, don't underestimate my laziness. I have it do the hard work and I just fill in where I need to or, in this case, I was stumped and I thought I'd see if it saw something in the JSON that I was missing. It seemed very much confused about the JSON issue.

HenryGeorgist commented 2 weeks ago

i sent you a file that worked for me in qgis after my edits. i use jsonlint.com to validate the json and ensure it meets the specification, it helps me point out the problems that need to be resolved. your description of geopackage seems consistent with the geojson writer in that it is not triggering the close method when the program is shutting down. looks like i need to fix the code one way or another eventually. i was just giving you a hard time about chat gpt.

jlgutenson commented 2 weeks ago

Thank you! My email changed. Did you use jlgutenson@gmail.com or joseph@follumhydro.com?

We just invested in the paid version of ChatGPT. We're somewhat hooked. It can be really good for working from home without a rubber duck handy! But it also can lead you on some wild goose chases!

I moved back to the original gdal issue that's plaguing v0.0.2. Still haven't figured that one out either. I'm not sure what GDAL version 2.8 dependencies there are in go-consequences.

It may be worth mentioning that this is all in an effort to demonstrate how go-consequences can be used to forecast flood consequences, if provided a forecasted flood depth grid.

HenryGeorgist commented 2 weeks ago

i run the latest in a different container and this is what i use as my production container:

FROM ghcr.io/osgeo/gdal:ubuntu-full-3.8.3 as builder

ENV TZ=America/New_York
ENV PATH=/go/bin:$PATH
ENV GOROOT=/go
ENV GOPATH=/src/go

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone &&\
    mkdir /go &&\
    mkdir -p /src/go &&\
    apt update &&\
    apt -y install build-essential &&\
    apt -y install pkg-config &&\
    apt -y install gdal-bin gdal-data libgdal-dev &&\
    apt -y install wget &&\
    wget https://golang.org/dl/go1.19.13.linux-amd64.tar.gz -P / &&\
    tar -xvzf /go1.19.13.linux-amd64.tar.gz -C / &&\
    apt -y install vim &&\
    apt -y install git

WORKDIR /app
RUN git clone https://github.com/USACE/consequences-runner.git
WORKDIR /app/consequences-runner
RUN go mod download
RUN go mod tidy
RUN go build 

FROM ghcr.io/osgeo/gdal:ubuntu-full-3.8.3 as prod
WORKDIR /app
COPY --from=builder /app/consequences-runner/consequences-runner .
CMD ["/app/consequences-runner"]

it uses the osgeo ubuntu full - which is not small and maybe is overkill.

HenryGeorgist commented 2 weeks ago

gah, but thats the same one you were using. which doesnt make sense

jlgutenson commented 2 weeks ago

I think I got the gdal issues in v0.0.2 solved (with the help of our friend ChatGPT :-))! Maybe you can confirm. I've attached a geotiff for easy access.

depth_max_clipped2.json

Here is the modified Dockerfile.prod. It's just tricky the image into thinking it has a libgdal.so.28 file with a symbolic link.

# Stage 1: Builder
FROM debian:bullseye AS builder

ENV TZ=America/New_York
ENV PATH=/go/bin:$PATH
ENV GOROOT=/go
ENV GOPATH=/src/go

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone &&\
    mkdir -p /go &&\
    mkdir -p /src/go &&\
    apt update &&\
    apt -y install build-essential &&\
    apt -y install libgdal-dev &&\
    apt -y install gdal-bin &&\
    apt -y install gdal-data &&\
    apt -y install wget &&\
    wget https://golang.org/dl/go1.19.5.linux-amd64.tar.gz -P / &&\
    tar -xvzf /go1.19.5.linux-amd64.tar.gz -C / &&\
    apt -y install vim &&\
    apt -y install git

WORKDIR /app
RUN git clone https://github.com/USACE/go-consequences.git
WORKDIR /app/go-consequences
RUN go mod download
RUN go mod tidy
RUN go build -o main .

# Stage 2: Production
FROM ghcr.io/osgeo/gdal:ubuntu-full-3.8.3 AS prod

# Move the executable from build to production
WORKDIR /app
COPY --from=builder /app/go-consequences/main ./

# Install necessary GDAL libraries
RUN apt update && \
    apt install -y libgdal-dev && \
    ln -s /usr/lib/libgdal.so /usr/lib/libgdal.so.28 && \
    ldconfig

# Verify library dependencies for debugging
RUN ldd ./main

ENTRYPOINT ["/app/main"]

I then built the Docker image: docker build -f Dockerfile.prod -t jlgutenson/go-consequences .

Then I can use the geojson to run go-consequences and output a geopackage:

docker run --rm --mount type=bind,source="C:\Users\jlgut\OneDrive\Desktop\go-consequences",target=/data jlgutenson/go-consequences /data/data/joseph_test.json

HenryGeorgist commented 2 weeks ago

that looks good to me if the results make sense and it runs!

jlgutenson commented 2 weeks ago

Sounds good, we will test a few different times and see if things make sense. Things seem correct for my Asheville test case.