emontgomery-usgs commented 8 years ago

@kwilcox, @rsignell-usgs Today I tried to convert 2 new files from the CALCSTWET project. Since it didn't have many files, I decided to re-convert the whole experiment.

First, I did git pull, then started the IOOS env. then I issued this command: python collect.py --projects CALCSTWET --download --output=../../CF-1.6

and got this error msg when things went south 2016-10-18 13:07:06 gam root[10420] INFO Translating /usgs/data2/emontgomery/stellwagen/usgs-cmg-portal/woods_hole_obs_data/download/CALCSTWET/9821ysi-a.nc into CF1.6 format: /usgs/data2/emontgomery/stellwagen/CF-1.6/CALCSTWET/9821ysi-a.nc python: /feedstock_root/build_artefacts/work/netcdf-c-4.4.1/libsrc/nc_hashmap.c:106: rehashVar: Assertion `count == hm->count' failed. Aborted (core dumped)

All the files got to "download" and have today's date, and the files in the output directory look good up to and including 9821ysi-a.nc. It did not make a new version of 9831HRaqds-a.nc, nor did it do the other new file 9841ysa-a.nc.

What went wrong this time? Thanks!

rsignell-usgs commented 8 years ago

@emontgomery-usgs, in response to this issue, I just tried upgrading pyaxiom and it upgraded from 1.2 to 1.3.2. You should see this if you type "conda list pyaxiom".
Perhaps try again to see if that helped?

emontgomery-usgs commented 8 years ago

@rsignell-usgs $ conda list pyaxiom packages in environment at /home/usgs/miniconda/envs/ioos: pyaxiom 1.3.2 py27_0 conda-forge $python collect.py --projects CALCSTWET --download --output=../../CF-1.6 runs until it generates the same answer as before. Didn't fix it- sorry!

kwilcox commented 7 years ago

Apologies... this slipped by my inbox! This was most likely a bug in conda from when I last ran the tests until you installed new packages. I just ran the tests from scratch and they work. Can you try again? If you get the same error can you paste the contents of conda list? Thanks!

emontgomery-usgs commented 7 years ago

Still broken when I run it- I did a git pull first too.

source activate ioos (ioos) emontgomery@gam :/usgs/data2/emontgomery/stellwagen/usgs-cmg-portal/woods_hole_obs_data$ conda list pyaxiom

packages in environment at /home/usgs/miniconda/envs/ioos:

# pyaxiom 1.3.2 py27_0 conda-forge (ioos) emontgomery@gam :/usgs/data2/emontgomery/stellwagen/usgs-cmg-portal/woods_hole_obs_data$ python collect.py --projects CALCSTWET --download --output=../../CF-1.6 2016-11-07 11:51:17 gam thredds_crawler[27236] INFO Crawling: http://geoport.whoi.edu/thredds/catalog/usgs/data2/emontgomery/stellwagen/Data/CALCSTWET/catalog.xml 2016-11-07 11:51:17 gam thredds_crawler[27236] INFO Ignoring dataset based on 'selects'. ID: usgs/data2/emontgomery/stellwagen/Data/CALCSTWET/9831HRaqdb-cal.nc 2016-11-07 11:51:17 gam root[27236] INFO Found 7 datasets in CALCSTWET!

The download part went as expected, and again it does the convert fine, but fails on the ysi file. below is the tail of what's dieplayed.

2016-11-07 11:51:30 gam root[27236] INFO Translating /usgs/data2/emontgomery/stellwagen/usgs-cmg-portal/woods_hole_obs_data/download/CALCSTWET/ 9811aqd-a.nc into CF1.6 format: /usgs/data2/emontgomery/stellwagen/CF-1.6/CALCSTWET/9811aqd-a.nc 2016-11-07 11:51:34 gam root[27236] DEBUG Could not find CF mapping for EPIC code 624.0 2016-11-07 11:51:34 gam root[27236] DEBUG Could not find CF mapping for EPIC code 624.0 2016-11-07 11:51:34 gam root[27236] DEBUG Could not find CF mapping for EPIC code 500.0 2016-11-07 11:51:34 gam root[27236] DEBUG Could not find CF mapping for EPIC code 502.0 2016-11-07 11:51:34 gam root[27236] DEBUG Could not find CF mapping for EPIC code 106.0 2016-11-07 11:51:34 gam root[27236] INFO Translating /usgs/data2/emontgomery/stellwagen/usgs-cmg-portal/woods_hole_obs_data/download/CALCSTWET/ 9821ysi-a.nc into CF1.6 format: /usgs/data2/emontgomery/stellwagen/CF-1.6/CALCSTWET/9821ysi-a.nc python: /feedstock_root/build_artefacts/work/netcdf-c-4.4.1/libsrc/nc_hashmap.c:106: rehashVar: Assertion `count == hm->count' failed. Aborted (core dumped)

On Mon, Nov 7, 2016 at 9:42 AM, Kyle Wilcox notifications@github.com wrote:

Apologies... this slipped by my inbox! This was most likely a bug in conda from when I last ran the tests until you installed new packages. I just ran the tests from scratch and they work. Can you try again? If you get the same error can you paste the contents of conda list? Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/227#issuecomment-258853347, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsHnu1oZUCKWQNaFuiz78o1L9tN3Vqkks5q7zjygaJpZM4KaD-M .

Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 457-2356

kwilcox commented 7 years ago

Can you please post the results of conda list while you have the ioos environment activated?

emontgomery-usgs commented 7 years ago

@kwilcox any progress on this (posted 2 days ago)? here you go. the results of conda list are attached.

On Mon, Nov 7, 2016 at 11:58 AM, Kyle Wilcox notifications@github.com wrote:

Can you please post the results of conda list while you have the ioos environment activated?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/227#issuecomment-258893746, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsHniNW50wCWwqKQtnV0t8LIhY5enIeks5q71jLgaJpZM4KaD-M .

Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 457-2356

source activate ioos (ioos) emontgomery@gam:~$ conda list

packages in environment at /home/usgs/miniconda/envs/ioos:

# nodejs 6.1.0 0 amfarrell anaconda-navigator 1.2.3 py27_0 beautifulsoup4 4.4.1 py27_0 bokeh 0.12.1 py27_0 boto3 1.3.1 py27_0 cffi 1.6.0 py27_0 affine 2.0.0.post1 py27_0 conda-forge alabaster 0.7.9 py27_0 conda-forge anaconda-client 1.5.1 py27_0 conda-forge babel 2.3.4 py27_0 conda-forge backports.shutil_get_terminal_size 1.0.0 py27_0 conda-forge backports_abc 0.4 py27_0 conda-forge biggus 0.14.0 py27_1 conda-forge blas 1.1 openblas conda-forge botocore 1.4.41 py27_0 conda-forge bottleneck 1.0.0 np111py27_0 conda-forge ca-certificates 2016.8.2 3 conda-forge cairo 1.12.18 8 conda-forge cartopy 0.14.2 np111py27_1 conda-forge cc-plugin-glider 1.0.1 py27_0 conda-forge certifi 2016.8.2 py27_0 conda-forge cf_units 1.1.1 py27_0 conda-forge click 6.6 py27_0 conda-forge click-plugins 1.0.3 py27_0 conda-forge cloudpickle 0.2.1 py27_2 conda-forge clyent 1.2.1 py27_0 conda-forge cmocean 1.0 py27_0 conda-forge colorama 0.3.7 py27_0 conda-forge coloredlogs 5.0 py27_1 conda-forge compliance-checker 2.2.0 py27_1 conda-forge configparser 3.5.0b2 py27_0 conda-forge ctd 0.2.1 py27_0 conda-forge curl 7.49.1 0 conda-forge cycler 0.10.0 py27_0 conda-forge cyordereddict 1.0.0 py27_1 conda-forge cython 0.24.1 py27_0 conda-forge dask 0.10.2 py27_0 conda-forge decorator 4.0.10 py27_0 conda-forge descartes 1.0.2 py27_0 conda-forge distributed 1.11.3 py27_0 conda-forge docopt 0.6.2 py27_0 conda-forge ecmwf_grib 1.15.0 0 conda-forge entrypoints 0.2.2 py27_0 conda-forge epic2cf 0.0.4 py27_0 conda-forge esmf 7.0.0 4 conda-forge expat 2.1.0 1 conda-forge fiona 1.7.0 np111py27_2 conda-forge folium 0.2.1 py27_0 conda-forge fontconfig 2.11.1 3 conda-forge freetype 2.6.3 1 conda-forge freexl 1.0.2 1 conda-forge funcsigs 1.0.2 py27_0 conda-forge functools32 3.2.3.2 py27_1 conda-forge gdal 2.1.1 np111py27_1 conda-forge geojson 1.3.3 py27_0 conda-forge geojsonio 0.0.2 py27_0 conda-forge geopandas 0.2.1 py27_0 conda-forge geopy 1.10.0 py27_0 conda-forge geos 3.4.2 2 conda-forge gettext 0.19.7 1 conda-forge giflib 5.1.2 1 conda-forge github3.py 1.0.0a4 py27_0 conda-forge glib 2.48.0 4 conda-forge gridgeo 0.1.0 py27_0 conda-forge gsl 2.1 2 conda-forge gsw 3.0.3 py27_0 conda-forge h5netcdf 0.2.2 py27_0 conda-forge h5py 2.6.0 np111py27_5 conda-forge hdf4 4.2.11 4 conda-forge hdf5 1.8.17 2 conda-forge humanfriendly 1.44.7 py27_2 conda-forge icu 56.1 4 conda-forge ipykernel 4.3.1 py27_1 conda-forge ipython 5.0.0 py27_0 conda-forge ipywidgets 5.2.2 py27_0 conda-forge iris 1.9.2 py27_1 conda-forge isodate 0.5.4 py27_0 conda-forge jasper 1.900.1 3 conda-forge jinja2 2.8 py27_1 conda-forge jmespath 0.9.0 py27_0 conda-forge jpeg 9b 0 conda-forge json-c 0.12 0 conda-forge jsonschema 2.5.1 py27_0 conda-forge jupyter_client 4.3.0 py27_0 conda-forge jupyter_console 5.0.0 py27_0 conda-forge jupyter_core 4.1.0 py27_0 conda-forge kealib 1.4.6 3 conda-forge libdap4 3.18.0 2 conda-forge libffi 3.2.1 2 conda-forge libiconv 1.14 3 conda-forge libmo_unpack 3.0 1 conda-forge libnetcdf 4.4.1 0 conda-forge libpng 1.6.23 0 conda-forge libpq 9.5.3 1 conda-forge libsodium 1.0.10 0 conda-forge libspatialindex 1.8.5 1 conda-forge libspatialite 4.3.0a 9 conda-forge libtiff 4.0.6 6 conda-forge libuuid 1.0.3 1 conda-forge libxml2 2.9.3 9 conda-forge locket 0.2.0 py27_0 conda-forge markupsafe 0.23 py27_0 conda-forge matplotlib 1.5.2 np111py27_6 conda-forge mistune 0.7.3 py27_0 conda-forge mo_pack 0.2.0 np111py27_0 conda-forge mock 2.0.0 py27_0 conda-forge mpld3 0.2 py27_0 conda-forge mplexporter 0.0.1 py27_0 conda-forge mplleaflet 0.0.5 py27_0 conda-forge mpmath 0.19 py27_0 conda-forge munch 2.0.4 py27_0 conda-forge nbconvert 4.2.0 py27_0 conda-forge nbformat 4.0.1 py27_0 conda-forge nco 4.6.1 0 conda-forge ncurses 5.9 8 conda-forge netcdf-fortran 4.4.4 1 conda-forge netcdf4 1.2.4 np111py27_2 conda-forge nose 1.3.7 py27_0 conda-forge notebook 4.2.2 py27_0 conda-forge numpy 1.11.1 py27_blas_openblas_201 [blas_openblas] conda-forge oceans 0.2.5 py27_0 conda-forge oct2py 3.5.9 py27_0 conda-forge openblas 0.2.18 4 conda-forge openjpeg 2.1.0 6 conda-forge openssl 1.0.2h 2 conda-forge owslib 0.13.0 py27_0 conda-forge paegan 1.1.4 py27_0 conda-forge pandas 0.18.1 np111py27_1 conda-forge pango 1.40.1 0 conda-forge partd 0.3.5 py27_0 conda-forge pathlib2 2.1.0 py27_0 conda-forge pbr 1.10.0 py27_0 conda-forge pcre 8.38 1 conda-forge pep8 1.7.0 py27_0 conda-forge pexpect 4.2.0 py27_1 conda-forge pickleshare 0.7.3 py27_0 conda-forge pillow 3.3.0 py27_0 conda-forge pip 8.1.2 py27_0 conda-forge pixman 0.32.6 0 conda-forge postgresql 9.5.3 2 conda-forge proj.4 4.9.2 0 conda-forge prompt_toolkit 1.0.4 py27_0 conda-forge protobuf 3.0.0b3 py27_0 conda-forge psutil 4.3.0 py27_0 conda-forge psycopg2 2.6.2 py27_1 conda-forge ptyprocess 0.5.1 py27_0 conda-forge pyaxiom 1.3.2 py27_0 conda-forge pyepsg 0.3.1 py27_0 conda-forge pyflakes 1.2.3 py27_0 conda-forge pygc 0.2 py27_0 conda-forge pygeoif 0.6 py27_1 conda-forge pygments 2.1.3 py27_0 conda-forge pyke 1.1.1 py27_0 conda-forge pyncml 0.0.8 py27_0 conda-forge pynco 0.0.2 py27_0 conda-forge pyoos 0.7.0 py27_0 conda-forge pyparsing 2.1.1 py27_0 conda-forge pyproj 1.9.5.1 py27_0 conda-forge pysal 1.11.2 py27_0 conda-forge pysgrid 0.1.1 py27_0 conda-forge pyshp 1.2.3 py27_0 conda-forge python-dateutil 2.5.3 py27_0 conda-forge python-ecmwf_grib 1.15.0 np111py27_0 conda-forge pytz 2016.6.1 py27_0 conda-forge pyugrid 0.1.6 py27_0 conda-forge pyyaml 3.11 py27_0 conda-forge pyzmq 15.4.0 py27_0 conda-forge qrcode 5.3 py27_0 conda-forge qt 4.8.7 0 conda-forge qtconsole 4.2.1 py27_0 conda-forge qtpy 1.1.1 py27_0 conda-forge rasterio 0.36.0 np111py27_2 conda-forge readline 6.2 0 conda-forge requests 2.10.0 py27_0 conda-forge rtree 0.8.2 py27_0 conda-forge s3fs 0.0.6 py27_0 conda-forge scikit-learn 0.17.1 np111py27_blas_openblas_201 [blas_openblas] conda-forge scipy 0.18.0 np111py27_blas_openblas_201 [blas_openblas] conda-forge seawater 3.3.4 py27_0 conda-forge setuptools 25.1.6 py27_0 conda-forge shapely 1.5.16 np111py27_1 conda-forge simplegeneric 0.8.1 py27_0 conda-forge singledispatch 3.4.0.3 py27_0 conda-forge sip 4.18 py27_0 conda-forge siphon 0.4.0 py27_0 conda-forge six 1.10.0 py27_0 conda-forge snowballstemmer 1.2.1 py27_0 conda-forge snuggs 1.4.0 py27_0 conda-forge sphinx 1.4.5 py27_0 conda-forge spyder 2.3.9 py27_0 conda-forge sqlite 3.13.0 1 conda-forge ssl_match_hostname 3.5.0.1 py27_0 conda-forge statsmodels 0.6.1 np111py27_0 conda-forge sympy 1.0 py27_0 conda-forge terminado 0.6 py27_0 conda-forge thredds_crawler 1.5.0 py27_0 conda-forge toolz 0.8.0 py27_0 conda-forge tornado 4.4.1 py27_1 conda-forge traitlets 4.2.2 py27_0 conda-forge udunits2 2.2.20 1 conda-forge uritemplate.py 0.3.0 py27_0 conda-forge vincent 0.4.4 py27_0 conda-forge wcwidth 0.1.7 py27_0 conda-forge wheel 0.29.0 py27_0 conda-forge widgetsnbextension 1.2.6 py27_3 conda-forge xarray 0.8.1 py27_0 conda-forge xerces-c 3.1.4 2 conda-forge xz 5.2.2 0 conda-forge yaml 0.1.6 0 conda-forge yaml2ncml 0.4.0 py27_0 conda-forge zeromq 4.1.5 0 conda-forge zlib 1.2.8 3 conda-forge cryptography 1.4 py27_0 docutils 0.12 py27_2 enum34 1.1.6 py27_0 fastcache 1.0.2 py27_1 futures 3.0.5 py27_0 harfbuzz 0.9.39 1 idna 2.1 py27_0 imagesize 0.7.1 py27_0 cligj 0.2.0 py27_0 ioos tardis 0.1.1 py27_2 ioos utilities 0.5.0 py27_0 ioos ipaddress 1.0.16 py27_0 ipython-notebook 4.0.4 py27_0 ipython_genutils 0.1.0 py27_0 jedi 0.9.0 py27_1 jupyter 1.0.0 py27_3 libgcc 5.2.0 0 libgfortran 3.0.0 1 libxslt 1.1.28 0 lxml 3.6.1 py27_0 msgpack-python 0.4.7 py27_0 ndg-httpsclient 0.3.3 py27_0 patsy 0.4.1 py27_0 pyasn1 0.1.9 py27_0 pycparser 2.14 py27_1 pyopenssl 0.14 py27_0 pyqt 4.11.4 py27_4 python 2.7.12 1 rope 0.9.4 py27_1 seaborn 0.7.1 py27_0 simplejson 3.8.2 py27_0 sqlalchemy 1.0.13 py27_0 tblib 1.3.0 py27_0 tk 8.5.18 0 xlrd 1.0.0 py27_0 (ioos) emontgomery@gam:~$

emontgomery-usgs commented 7 years ago

@kwilcox @rsignell-usgs - I tried to convert the files in CALCSTWET to CF again today, and it still doesn't work- same core dump error. Would one of you please do the conversion, so the data is available in CF.

Today I added a new dataset: SANDWICH2016. There's a pull request for the needed mods to the .csv file. Once that is ready (hopefully tomorrow), I'll try to convert these files. If it also fails, perhaps one of you can do the conversion.

Once they have been converted to CF, could the be ingested into the portal? Thanks!

emontgomery-usgs commented 7 years ago

@rsignell-usgs @kwilcox

I had hoped this problem was something abberant in the CALCSTWET data, but no. I tried running the new dataset I put up, and that failed. Then I went back and tried convert on one I'd already converted with a previous version.

Here are the error msgs: == this is the new dataset python collect.py --projects SANDWICH2016 --download --output=../../CF-1.6/SANDWICH2016 Translating /usgs/data2/emontgomery/stellwagen/usgs-cmg-portal/woods_hole_obs_data/download/SANDWICH2016/10591dwvs-cal.nc into CF1.6 format: /usgs/data2/emontgomery/stellwagen/CF-1.6/SANDWICH2016/SANDWICH2016/10591dwvs-cal.nc python: /feedstock_root/build_artefacts/work/netcdf-c-4.4.1/libsrc/nc_hashmap.c:106: rehashVar: Assertion `count == hm->count' failed. Aborted (core dumped)

== this is the one that had previously been converted python collect.py --projects RCNWR --download --output=../../CF-1.6/tmp_RC python: /feedstock_root/build_artefacts/work/netcdf-c-4.4.1/libsrc/nc_hashmap.c:106: rehashVar: Assertion `count == hm->count' failed. Aborted (core dumped)

Please help identify what's wrong on geoport. Thanks!

kwilcox commented 7 years ago

I'm running them right now with no problems... @rsignell-usgs can you re-create the environment from scratch in a few minutes? Going to push something first...

rsignell-usgs commented 7 years ago

Okay, @ocefpaf to the rescue again: "It seems that there was a bug in libnetcdf that was fixed in the latest version. b/c we are still using that crappy channel config (by adding ioos too) we were getting an outdated version. I updated it on IOOS and things seems to be working".

So @emontgomery-usgs , this time I've made a new environment called "CONVERT-WH" instead of using the IOOS environment. I should have done this a long time ago. So on geoport, please try:

source activate CONVERT-WH
cd /usgs/data2/emontgomery/stellwagen/usgs-cmg-portal/woods_hole_obs_data
python collect.py --projects SANDWICH2016 --download --output=../../CF-1.6/SANDWICH2016

emontgomery-usgs commented 7 years ago

emontgomery@gam:~$ source activate CONVERT-WH Error: no such directory: /opt/wakari/anaconda/envs/CONVERT-WH/bin

Is this where it's supposed to look? I don't see envs under anaconda... $ ls /opt/wakari/anaconda bin etc LICENSE_Accelerate.txt plugins conda-meta Examples LICENSE_IOPro.txt share config include LICENSE_MKL_Optimizations.txt ssl docs lib pkgs

On Mon, Dec 5, 2016 at 1:02 PM, Rich Signell notifications@github.com wrote:

Okay, @ocefpaf https://github.com/ocefpaf to the rescue again: "It seems that there was a bug in libnetcdf that was fixed in the latest version. b/c we are still using that crappy channel config (by adding ioos too) we were getting an outdated version. I updated it on IOOS and things seems to be working".

So @emontgomery-usgs https://github.com/emontgomery-usgs , this time I've made a new environment called "CONVERT-WH" instead of using the IOOS environment. I should have done this a long time ago. So on geoport, please try:

source activate CONVERT-WH cd /usgs/data2/emontgomery/stellwagen/usgs-cmg-portal/woods_hole_obs_data python collect.py --projects SANDWICH2016 --download --output=../../CF-1.6/SANDWICH2016

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/227#issuecomment-264928052, or mute the thread https://github.com/notifications/unsubscribe-auth/ABsHngFdjmwNXmH5CPZCOyXXQB5lzvmmks5rFFHMgaJpZM4KaD-M .

-- Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 457-2356

emontgomery-usgs commented 7 years ago

OK- I've got the activation of CONVERT-WH working, and both the CALCSTWET and SANDWICH2016 datasets converted successfully.

Thank you!

USGS-CMG / usgs-cmg-portal

collect.py failure with time series addition #227

packages in environment at /home/usgs/miniconda/envs/ioos:

packages in environment at /home/usgs/miniconda/envs/ioos: