piyushrpt
commented
5 years ago How much RAM are you working with?
Closed cmarshak closed 4 years ago
piyushrpt
commented
5 years ago How much RAM are you working with?
piyushrpt
commented
5 years ago If you are building on top of a docker image that already has packages installed via conda, I would be careful about calling conda install again. A more robust way would be to gather list of all conda packages from your parent docker images and include all of it in a single requirements.txt file; and use that for installing. I believe the chain is
jupyter/minimal-notebook (no conda) -> jupyter/scipy-notebook (conda python)->jupyter/datascience-notebook (conda r)
If you derive from datascience-notebook, you can potentially introduce conflicts between libraries installed by scipy-notebook and your command. Just list those packages and your requirements in a single file and use it - conda will try to resolve any library conflicts if it sees all the packages it needs to install in one place.
cmarshak
commented
5 years ago In regards to your first suggestion regarding RAM, I increased the docker memory available from 2 GiB (default) to 10 GiB. No luck - same illegal instruction.
Unfortunately, changing the docker file to:
FROM jupyter/minimal-notebook
RUN /bin/bash -c "conda install -c conda-forge isce2 scikit-image scikit-learn rasterio geopandas pandas"actually makes the notebook crash even more quickly - opening up a new notebook and running import numpy as np kills the kernel for example.
I will try using miniconda images, but makes opening jupyter notebooks a little more complicated.
piyushrpt
commented
5 years ago You can also try the Dockerfile provided within the isce2 repo. And just use it with docker run by mounting folder containing data to it and executing stripmapApp. That way you wont have to install jupyter within the image. You can explore the results with Jupyter on your own machine.
cmarshak
commented
5 years ago Well, I was unsuccessful in installing isce with a miniconda docker image.
I was able to install isce with the dockerfile
FROM continuumio/miniconda
RUN /bin/bash -c "conda install -c conda-forge isce2 scikit-image scikit-learn matplotlib rasterio geopandas pandas jupyter"on Friday, June 28th, but no luck with my desktop the following Monday or since. I am pretty sure it is an issue with conda-forge builds and isce. I tried numerous ways with miniconda including the omission of all packages except isce2 as in a dockerfile:
FROM continuumio/miniconda
RUN /bin/bash -c "conda install -c conda-forge isce2"but have been encountering the same illegal instruction on the topo step. I also tried importing the docker image onto my desktop from my laptop using an environment.yml file (had to use --no-builds) and loading a tar directly. The latter led to package conflicts. If there is a way to update the isce2 conda formula that might fix this for me.
I did try using the dockerfile on the isce2 repository. I think this is identical to the isce/isce2 image on dockerhub.
For both of these images I had to append the file /opt/isce2/isce_env.sh to the .bashrc (or run the isce_env.sh) so the applications would work, which are located in /opt/isce2.
I added the jupyter, scikit-image, rasterio installs at the end of the dockerfile in the isce repo for installation and it currently works just fine.
Let me know if and when the linux conda installation is fixed. Would really like to start using that. But even better would be the macos conda build :)
Thank you for your continued help.
piyushrpt
commented
5 years ago It is still unclear as to what is causing this issue. If you can exactly figure out the problem, we can update the recipe with a solution. Seems like the issue might be related to using the miniconda image as the base.
piyushrpt
commented
5 years ago See https://forums.docker.com/t/certain-conda-packages-make-python-crash-with-illegal-instruction-core-dumped/70718 . Try running the apps using gdb as shown in this thread: https://github.com/numpy/numpy/issues/11517
Might provide more information than just an illegal instruction.
cmarshak
commented
5 years ago I am pretty sure it is an openblas issue. On my laptop (where isce works with topo), I have the openblas version 0.3.5, but I am not sure how this was downloaded since it doesn't seem like this is available.
The gdb doesn't seem to be any more informative. Using the docker image created as:
FROM continuumio/miniconda
RUN conda install -c conda-forge isce2
RUN apt-get install gdbproduces
$ gdb python
(gdb) run /opt/conda/lib/python3.7/site-packages/isce/applications/stripmapApp.py stripmapApp.xml --start=topo --end=topo
Starting program: /opt/conda/bin/python /opt/conda/lib/python3.7/site-packages/isce/applications/stripmapApp.py stripmapApp.xml --start=topo --end=topo
warning: Error disabling address space randomization: Operation not permitted
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
OpenCV2 does not appear to be installed / is not importable.
OpenCV2 is needed for this step. You may experience failures ...
2019-07-08 22:51:30,250 - isce.insar - INFO - ISCE VERSION = 2.3, RELEASE_SVN_REVISION = 2531,RELEASE_DATE = 20190112, CURRENT_SVN_REVISION = Unknown
ISCE VERSION = 2.3, RELEASE_SVN_REVISION = 2531,RELEASE_DATE = 20190112, CURRENT_SVN_REVISION = Unknown
2019-07-08 22:51:30,260 - root - INFO - master sensor not provided explicitly, using common sensor
2019-07-08 22:51:30,291 - root - INFO - slave sensor not provided explicitly, using common sensor
do_unwrap True
Step processing
Running step topo
2019-07-08 22:51:30,492 - isce.insar.runTopo - INFO - Running topo
2019-07-08 22:51:30,494 - isce.insar.runTopo - INFO - Geometry directory geometry already exists.
Polynomial Order: 0 - by - 2
6.65625 -0.0102682 1.29377e-06
Using native doppler information for bbox estimation
Polynomial Order: 0 - by - 2
-7.3064 -0.00776838 9.54456e-07
Using native doppler information for bbox estimation
Polynomial Order: 0 - by - 2
6.65625 -0.0102682 1.29377e-06
2019-07-08 22:51:31,558 - isce.zerodop.topozero - WARNING - Default Peg heading set to: -2.928317862178095
Polynomial Order: 0 - by - 1
948796 9.36851
GDAL open (R): demLat_N18_N21_Lon_W156_W154.dem.wgs84.vrt
API open (WR): geometry/lat.rdr.full
API open (WR): geometry/lon.rdr.full
API open (WR): geometry/z.rdr.full
API open (WR): geometry/los.rdr.full
C pointer already created. Finalize and recreate if image dimensions changed.
C pointer already created. Finalize and recreate if image dimensions changed.
Initializing biquintic interpolator
Orbit interpolation method: hermite
Max threads used: 4
DEM parameters:
Dimensions: 7200 10800
Top Left: -156.00000000000000 21.000000000000000
Spacing: 2.7777777777777778E-004 -2.7777777777777778E-004
Lon: -156.00000000000000 -154.00027777777777
Lat: 18.000277777777779 21.000000000000000
Estimated DEM bounds needed for global height range:
Lon: -155.61269408623309 -154.40615106325461
Lat: 18.858761900799546 20.156668160939397
Actual DEM bounds used:
Dimensions: 4345 4674
Top Left: -155.61277777777778 20.156944444444445
Spacing: 2.7777777777777778E-004 -2.7777777777777778E-004
Lon: -155.61277777777778 -154.40611111111110
Lat: 18.858888888888888 20.156944444444445
Lines: 3036 7709
Pixels: 1395 5739
Max DEM height: 4238.00000
Primary iterations: 25
Secondary iterations: 10
Distance threshold : 5.0000000000000003E-002
Processing line: 1 7598.0494211415598
Dopplers: 6.6562538461180463 -11.890571484633030 -10.435713785312423
During startup program terminated with signal SIGILL, Illegal instruction.Is there something else I should try?
cmarshak
commented
5 years ago This is the environment.yml file for the working version on my laptop with builds (and extra packages, sorry):
channels:
- conda-forge
- defaults
dependencies:
- affine=2.2.2=py_0
- asn1crypto=0.24.0=py37_1003
- attrs=19.1.0=py_0
- backcall=0.1.0=py_0
- bleach=3.1.0=py_0
- boost-cpp=1.70.0=ha2d47e9_0
- bzip2=1.0.6=h14c3975_1002
- ca-certificates=2019.6.16=hecc5488_0
- cairo=1.16.0=h18b612c_1001
- certifi=2019.6.16=py37_0
- cffi=1.12.3=py37h8022711_0
- chardet=3.0.4=py37_1003
- click=7.0=py_0
- click-plugins=1.1.1=py_0
- cligj=0.5.0=py_0
- cloudpickle=1.2.1=py_0
- conda=4.7.5=py37_0
- conda-package-handling=1.3.10=py37_0
- cryptography=2.7=py37h72c5cf5_0
- curl=7.64.1=hf8cf82a_0
- cycler=0.10.0=py_1
- cytoolz=0.9.0.1=py37h14c3975_1001
- dask-core=2.0.0=py_0
- dbus=1.13.6=he372182_0
- decorator=4.4.0=py_0
- defusedxml=0.5.0=py_1
- deprecated=1.2.5=py_0
- descartes=1.1.0=py_3
- entrypoints=0.3=py37_1000
- expat=2.2.5=he1b5a44_1003
- ffmpeg=4.1.3=h167e202_0
- fftw=3.3.8=nompi_h7f3a6c3_1106
- fiona=1.8.6=py37hf242f0b_3
- fontconfig=2.13.1=he4413a7_1000
- freetype=2.10.0=he983fc9_0
- freexl=1.0.5=h14c3975_1002
- gdal=2.4.1=py37hf242f0b_0
- geopandas=0.5.0=py_2
- geos=3.7.1=hf484d3e_1000
- geotiff=1.4.3=h1105359_1000
- gettext=0.19.8.1=hc5be6a0_1002
- giflib=5.1.7=h516909a_1
- glib=2.58.3=h6f030ca_1001
- gmp=6.1.2=hf484d3e_1000
- gnutls=3.6.5=hd3a4fd2_1002
- graphite2=1.3.13=hf484d3e_1000
- gst-plugins-base=1.14.5=h0935bb2_0
- gstreamer=1.14.5=h36ae1b5_0
- h5py=2.9.0=nompi_py37hf008753_1102
- harfbuzz=2.4.0=h37c48d4_1
- hdf4=4.2.13=h9a582f1_1002
- hdf5=1.10.4=nompi_h3c11f04_1106
- icu=58.2=hf484d3e_1000
- idna=2.8=py37_1000
- imageio=2.5.0=py37_0
- ipykernel=5.1.1=py37h24bf2e0_0
- ipython=7.5.0=py37h24bf2e0_0
- ipython_genutils=0.2.0=py_1
- ipywidgets=7.4.2=py_0
- isce2=2.3.1=py37h55a01e5_1
- jasper=1.900.1=h07fcdf6_1006
- jedi=0.14.0=py37_0
- jinja2=2.10.1=py_0
- joblib=0.13.2=py_0
- jpeg=9c=h14c3975_1001
- json-c=0.13.1=h14c3975_1001
- jsonschema=3.0.1=py37_0
- jupyter=1.0.0=py_2
- jupyter_client=5.2.4=py_3
- jupyter_console=6.0.0=py_0
- jupyter_core=4.4.0=py_0
- kealib=1.4.10=h1978553_1003
- kiwisolver=1.1.0=py37hc9558a2_0
- krb5=1.16.3=h05b26f9_1001
- lame=3.100=h14c3975_1001
- libarchive=3.3.3=hb44662c_1005
- libblas=3.8.0=7_openblas
- libcblas=3.8.0=7_openblas
- libcurl=7.64.1=hda55be3_0
- libdap4=3.19.1=hd48c02d_1000
- libedit=3.1.20181209=hc058e9b_0
- libffi=3.2.1=hd88cf55_4
- libgcc-ng=8.2.0=hdf63c60_1
- libgdal=2.4.1=hdb8f723_0
- libgfortran-ng=7.3.0=hdf63c60_0
- libiconv=1.15=h516909a_1005
- libkml=1.3.0=h4fcabce_1010
- liblapack=3.8.0=7_openblas
- liblapacke=3.8.0=7_openblas
- libnetcdf=4.6.2=hbdf4f91_1001
- libpng=1.6.37=hed695b0_0
- libpq=11.3=h4e4e079_0
- libsodium=1.0.16=h14c3975_1001
- libspatialindex=1.9.0=he1b5a44_1
- libspatialite=4.3.0a=hb5ec416_1026
- libssh2=1.8.2=h22169c7_2
- libstdcxx-ng=8.2.0=hdf63c60_1
- libtiff=4.0.10=h57b8799_1003
- libuuid=2.32.1=h14c3975_1000
- libwebp=1.0.2=h99fbfcb_2
- libxcb=1.13=h14c3975_1002
- libxml2=2.9.9=h13577e0_0
- lz4-c=1.8.3=he1b5a44_1001
- lzo=2.10=h14c3975_1000
- mapclassify=2.1.0=py_0
- markupsafe=1.1.1=py37h14c3975_0
- matplotlib=3.1.0=py37_1
- matplotlib-base=3.1.0=py37hfd891ef_1
- mistune=0.8.4=py37h14c3975_1000
- munch=2.3.2=py_0
- nbconvert=5.5.0=py_0
- nbformat=4.4.0=py_1
- ncurses=6.1=he6710b0_1
- nettle=3.4.1=h1bed415_1002
- networkx=2.3=py_0
- notebook=5.7.8=py37_1
- numpy=1.16.4=py37h95a1406_0
- olefile=0.46=py_0
- openblas=0.3.5=h9ac9557_1001
- opencv=4.1.0=py37h5517eff_4
- openh264=1.8.0=hdbcaa40_1000
- openjpeg=2.3.1=h58a6597_0
- openmotif=2.3.7=0
- openssl=1.1.1b=h14c3975_1
- pandas=0.24.2=py37hb3f55d8_0
- pandoc=2.7.3=0
- pandocfilters=1.4.2=py_1
- parso=0.5.0=py_0
- pcre=8.41=hf484d3e_1003
- pexpect=4.7.0=py37_0
- pickleshare=0.7.5=py37_1000
- pillow=6.0.0=py37he7afcd5_0
- pip=19.1.1=py37_0
- pixman=0.38.0=h516909a_1003
- poppler=0.67.0=h2ad7f00_6
- poppler-data=0.4.9=1
- postgresql=11.3=hbeee2d4_0
- proj4=5.2.0=he1b5a44_1003
- prometheus_client=0.7.1=py_0
- prompt_toolkit=2.0.9=py_0
- pthread-stubs=0.4=h14c3975_1001
- ptyprocess=0.6.0=py_1001
- pycosat=0.6.3=py37h14c3975_1001
- pycparser=2.19=py37_1
- pygments=2.4.2=py_0
- pyopenssl=19.0.0=py37_0
- pyparsing=2.4.0=py_0
- pyproj=1.9.6=py37h516909a_1002
- pyqt=5.9.2=py37hcca6a23_0
- pyrsistent=0.15.2=py37h516909a_0
- pysocks=1.7.0=py37_0
- python=3.7.3=h5b0a415_0
- python-dateutil=2.8.0=py_0
- python-libarchive-c=2.8=py37_1004
- pytz=2019.1=py_0
- pywavelets=1.0.3=py37hd352d35_1
- pyzmq=18.0.2=py37hc4ba49a_0
- qt=5.9.7=h52cfd70_2
- qtconsole=4.5.1=py_0
- rasterio=1.0.24=py37hdff7cfa_0
- readline=7.0=h7b6447c_5
- requests=2.22.0=py37_0
- rtree=0.8.3=py37h666c49c_1002
- ruamel_yaml=0.15.71=py37h14c3975_1000
- scikit-image=0.15.0=py37hb3f55d8_2
- scikit-learn=0.21.2=py37hcdab131_1
- scipy=1.3.0=py37h921218d_0
- send2trash=1.5.0=py_0
- setuptools=41.0.1=py37_0
- shapely=1.6.4=py37h06cd6f9_1005
- sip=4.19.8=py37hf484d3e_1000
- six=1.12.0=py37_1000
- snuggs=1.4.6=py_0
- sqlite=3.28.0=h8b20d00_0
- terminado=0.8.2=py37_0
- testpath=0.4.2=py_1001
- tk=8.6.9=hed695b0_1002
- toolz=0.9.0=py_1
- tornado=6.0.3=py37h516909a_0
- tqdm=4.32.2=py_0
- traitlets=4.3.2=py37_1000
- tzcode=2019a=h516909a_1002
- urllib3=1.24.3=py37_0
- wcwidth=0.1.7=py_1
- webencodings=0.5.1=py_1
- wheel=0.33.4=py37_0
- widgetsnbextension=3.4.2=py37_1000
- wrapt=1.11.2=py37h516909a_0
- x264=1!152.20180806=h14c3975_0
- xerces-c=3.2.2=hac72e42_1001
- xorg-kbproto=1.0.7=h14c3975_1002
- xorg-libice=1.0.9=h516909a_1004
- xorg-libsm=1.2.3=h84519dc_1000
- xorg-libx11=1.6.7=h14c3975_1000
- xorg-libxau=1.0.9=h14c3975_0
- xorg-libxdmcp=1.1.3=h516909a_0
- xorg-libxext=1.3.4=h516909a_0
- xorg-libxft=2.3.3=h71203ad_0
- xorg-libxmu=1.1.3=h516909a_0
- xorg-libxrender=0.9.10=h516909a_1002
- xorg-libxt=1.1.5=h516909a_1003
- xorg-renderproto=0.11.1=h14c3975_1002
- xorg-xextproto=7.3.0=h14c3975_1002
- xorg-xproto=7.0.31=h14c3975_1007
- xz=5.2.4=h14c3975_1001
- yaml=0.1.7=had09818_2
- zeromq=4.3.1=hf484d3e_1000
- zlib=1.2.11=h7b6447c_3
- zstd=1.4.0=h3b9ef0a_0
prefix: /opt/condaWhen I try to build this in docker on my desktop (where it doesn't work), I get the following error:
ResolvePackageNotFound:
- libcblas==3.8.0=7_openblas -> libblas==3.8.0=7_openblas -> openblas=0.3.5
- openblas==0.3.5=h9ac9557_1001
- liblapack==3.8.0=7_openblas -> libblas==3.8.0=7_openblas -> openblas=0.3.5
- liblapacke==3.8.0=7_openblas -> libcblas==3.8.0=7_openblas -> libblas==3.8.0=7_openblas -> openblas=0.3.5
- libblas==3.8.0=7_openblas -> openblas=0.3.5When I try using just the versions (that is --no-builds using the conda), openblas is the sole version not available.
I include the following in my dockerfile to try to build:
COPY environment.yml /home/files/environment.yml
RUN conda env create -f /home/files/environment.ymlIs your laptop osx? I would not be surprised if you using an environment.yml from osx would work as is on linux. You might have a better shot at reproducing environments using environment.yml as long as the source / target are same - osx or x86_64 etc.
cmarshak
commented
5 years ago I am not totally sure if I understand what you mean by osx and x86_64 etc as source / target. Both my laptop (2012 macbook with High Sierra) and desktop (newer iMac with Mojave) are Mac. Not sure about there architectures except they are both 64 bit. Is there a fix for this with docker / conda?
Not sure if this is helpful, but below is the output of docker image inspect isce where the isce refers to the image FROM continuumio/miniconda, RUN conda install -c conda-forge isce2
[
{
"Id": "sha256:68f2ef6e111c1b8c45be0f9749b0e9e9f72febc37abe64c67729414c1c4d611b",
"RepoTags": [
"isce-test:latest"
],
"RepoDigests": [],
"Parent": "sha256:e3b0fcc9f628d902b7b59edf11c70b7f06ff58c41a8d404bac9d31c98913b166",
"Comment": "",
"Created": "2019-07-08T22:39:45.337910054Z",
"Container": "c3c79bd172603f17dfcc13142c3c4e8cb9b0a252e5a986be7885e8b530e1064e",
"ContainerConfig": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"LC_ALL=C.UTF-8"
],
"Cmd": [
"/bin/sh",
"-c",
"conda install -c conda-forge isce2"
],
"ArgsEscaped": true,
"Image": "sha256:e3b0fcc9f628d902b7b59edf11c70b7f06ff58c41a8d404bac9d31c98913b166",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"DockerVersion": "18.09.2",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"LC_ALL=C.UTF-8"
],
"Cmd": [
"/bin/bash"
],
"ArgsEscaped": true,
"Image": "sha256:e3b0fcc9f628d902b7b59edf11c70b7f06ff58c41a8d404bac9d31c98913b166",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/usr/bin/tini",
"--"
],
"OnBuild": null,
"Labels": null
},
"Architecture": "amd64",
"Os": "linux",
"Size": 3554431012,
"VirtualSize": 3554431012,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/095d146388b078c03458a1d094020a73bcd6dfddb2a867dcebb75cf726aa6fdb/diff:/var/lib/docker/overlay2/bab9e27d7768a97d9c4dd2f3f5925a5303a235079e1becfbf5d69a17fde4b34a/diff:/var/lib/docker/overlay2/2af0b4e05aa45811e38c02d2c16df68a64ba5237dac3112c12e54fd93187e328/diff:/var/lib/docker/overlay2/d4aabd565c1ec951938f87af8770cb9077b8b5338f0e3f47347af557d45fafc1/diff",
"MergedDir": "/var/lib/docker/overlay2/d170190c5f0df3b724d97c94c09249f11e852c56f941d138f0d4772037334d4b/merged",
"UpperDir": "/var/lib/docker/overlay2/d170190c5f0df3b724d97c94c09249f11e852c56f941d138f0d4772037334d4b/diff",
"WorkDir": "/var/lib/docker/overlay2/d170190c5f0df3b724d97c94c09249f11e852c56f941d138f0d4772037334d4b/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:f94641f1fe1f5c42c325652bf55f0513c881c86b620b912b15460e0bca07cc12",
"sha256:0c6784363557c6912964594137c559200cce1830725889c90cb8a14a217c0d41",
"sha256:6fc8e89571926c069c6af72e07e67c9f6487f54397af5c1f5188c198f66a55bb",
"sha256:52071325881f506e9738de9368909f2bed00a9ec579d6521f4b8f35efe584d7e",
"sha256:254c21f5b12ac7a500c0a20a31bcdfa9519c1a4a48e2146412eb6f7c42fe96c3"
]
},
"Metadata": {
"LastTagTime": "2019-07-08T22:39:46.783923363Z"
}
}
]Have you tried: https://hub.docker.com/r/isce/isce2
cmarshak
commented
5 years ago Yes - it works fine. It seems to be the same build as the docker on the github page.
For both these images, I had to run /opt/isce2/isce_env.sh for isce to work correctly as I mention above.
cmarshak
commented
4 years ago Well, the tutorial works fine now (without docker) on Mac using conda install -c conda-forge isce2 jupyter wget matplotlib.
Thank you!
Hi @piyushrpt and isce team,
I was trying to get the stripmap notebook to work.
I am currently using the linux conda build within a docker container for jupyter per the suggestion here. Here is my docker file for reference:
I am new to using docker to manage dependencies, so I just appended the path for the conda installation of the
isce/applicationsfor the notebook tutorial - sorry it's messy.In essence, when I get to the topo part of the notebook, I get an Illegal instruction and the geometry files don't write correctly. Here is the more complete story.
I believe everything up to the verifyDEM is working correctly.
When I run the cell below, I see the DEM verify throws some warning - it's hard for me to tell what's needed for topo.
Then: