cf-units=0.1.1 or cf_units=1.1

[rsignell-usgs]

I'm having trouble running the compliance checker on win64

It's complaining about cf-units=0.1.1 not being installed, but I've got cf_units=1.1 installed from the IOOS channel.

[ocefpaf]

I thought this was solved by #191. Can you check what version you have installed?

[rsignell-usgs]

[IOOS] C:\Users\rsignell>conda list compliance-checker
# packages in environment at C:\Users\rsignell\AppData\Local\Continuum\Miniconda
2\envs\IOOS:
#
compliance-checker        2.0.0                    py27_3    ioos

[rsignell-usgs]

Is there a "dash" vs. "underscore" problem here? The package is cf_units, not cf-units: http://anaconda.org/ioos/cf_units

[daf]

My compliance-checker conda env appears to have both installed:

cf-units                  1.0.0                     <pip>
cf_units                  1.0.0                    py27_0    ioos

The pip installed one is likely me being frustrated with the same problem at some point. I think you've found the issue, @rsignell-usgs .

[ocefpaf]

My compliance-checker conda env appears to have both installed:

cf-units 1.0.0 cf_units 1.0.0 py27_0 ioos

Something is broken! I cannot find the cf-units, only the correct cf_units.

https://pypi.python.org/pypi?%3Aaction=search&term=cf_units&submit=search

The pip installed one is likely me being frustrated with the same problem at some point. I think you've found the issue, @rsignell-usgs .

I will take a closer look at this next week.

[daf]

I'm also not seeing any ref in code/supporting to cf-units with the dash (in compliance checker base, anyway).

[rsignell-usgs]

My compliance checker env has these installed:

[IOOS] C:\Users\rsignell>conda list units

cf_units                  1.1                      py27_0    ioos
udunits2                  2.2.20                        0    ioos

[lukecampbell]

(test)(work)➜  site-packages  grep -Rni 'cf-units' ./
.//cf_units-1.1-py2.7.egg-info/PKG-INFO:2:Name: cf-units

Something somewhere is calling this package cf-units

[lukecampbell]

An absolutely fresh install of cf_units builds into cf-units:

(cf)➜  cf_units git:(cc26e37) python setup.py install --single-version-externally-managed --record record.txt
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/cf_units
copying cf_units/__init__.py -> build/lib/cf_units
copying cf_units/_version.py -> build/lib/cf_units
copying cf_units/config.py -> build/lib/cf_units
copying cf_units/util.py -> build/lib/cf_units
creating build/lib/cf_units/tests
copying cf_units/tests/__init__.py -> build/lib/cf_units/tests
copying cf_units/tests/test_coding_standards.py -> build/lib/cf_units/tests
copying cf_units/tests/test_unit.py -> build/lib/cf_units/tests
creating build/lib/cf_units/etc
copying cf_units/etc/site.cfg.template -> build/lib/cf_units/etc
UPDATING build/lib/cf_units/_version.py
set build/lib/cf_units/_version.py to '1.1'
running install_lib
creating /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units
copying build/lib/cf_units/__init__.py -> /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units
copying build/lib/cf_units/_version.py -> /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units
copying build/lib/cf_units/config.py -> /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units
creating /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/etc
copying build/lib/cf_units/etc/site.cfg.template -> /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/etc
creating /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/tests
copying build/lib/cf_units/tests/__init__.py -> /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/tests
copying build/lib/cf_units/tests/test_coding_standards.py -> /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/tests
copying build/lib/cf_units/tests/test_unit.py -> /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/tests
copying build/lib/cf_units/util.py -> /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units
byte-compiling /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/__init__.py to __init__.pyc
byte-compiling /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/_version.py to _version.pyc
byte-compiling /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/config.py to config.pyc
byte-compiling /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/tests/__init__.py to __init__.pyc
byte-compiling /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/tests/test_coding_standards.py to test_coding_standards.pyc
byte-compiling /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/tests/test_unit.py to test_unit.pyc
byte-compiling /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units/util.py to util.pyc
running install_data
creating /Users/lcampbell/.virtualenvs/cf/cf_units
copying COPYING -> /Users/lcampbell/.virtualenvs/cf/cf_units
copying COPYING.LESSER -> /Users/lcampbell/.virtualenvs/cf/cf_units
running install_egg_info
running egg_info
creating cf_units.egg-info
writing cf_units.egg-info/PKG-INFO
writing top-level names to cf_units.egg-info/top_level.txt
writing dependency_links to cf_units.egg-info/dependency_links.txt
writing manifest file 'cf_units.egg-info/SOURCES.txt'
reading manifest file 'cf_units.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'cf_units.egg-info/SOURCES.txt'
Copying cf_units.egg-info to /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units-1.1-py2.7.egg-info
running install_scripts
writing list of installed files to 'record.txt'
(cf)➜  cf_units git:(cc26e37) cd /Users/lcampbell/.virtualenvs/cf/lib/python2.7/site-packages/cf_units-1.1-py2.7.egg-info
(cf)➜  cf_units-1.1-py2.7.egg-info  ls
PKG-INFO             SOURCES.txt          dependency_links.txt top_level.txt
(cf)➜  cf_units-1.1-py2.7.egg-info  cat PKG-INFO
Metadata-Version: 1.0
Name: cf-units
Version: 1.1
Summary: UNKNOWN
Home-page: https://github.com/SciTools/cf_units
Author: UK Met Office
Author-email: UNKNOWN
License: UNKNOWN
Description: UNKNOWN
Platform: UNKNOWN

[rsignell-usgs]

Why are we building cf_units? We already have the conda package in the IOOS channel.

[lukecampbell]

I'm trying to find out why the package is being misnamed by following how conda builds it.

[daf]

Weirder, when I do python setup.py sdist, it'll create a cf_units.egg_info/PKG-INFO which has the cf-units incorrect string, but the tar.gz file it creates' PKG-INFO is correctly cf_units.

[lukecampbell]

I tried with a variety of strings, I see it changing any underscores into dashes...

[rsignell-usgs]

@lukecampbell , sorry my bad. :confounded:

[rsignell-usgs]

Maybe we should rename this package "cf-units"

[lukecampbell]

http://pybuilder.github.io/documentation/external_plugins.html#.VsX1U5MrLao

[lukecampbell]

In that case, you will have to set the name of your package (for PyPI) to pybuilder-* (dashes instead of underscores).

Why this?

Using a project name with dashes will make your plugin installable with pip.

• Pip replaces underscores with dashes, so if you use underscores in the project name it will not work.
• Having a package name with underscores is required, because using dashes in an package name is a syntax error at import.

[lukecampbell]

yeah so I think they need to change the name attribute to 'cf-units'

[rsignell-usgs]

@pelson?

[ocefpaf]

Why are we building cf_units? We already have the conda package in the IOOS channel.

Rich, Luke is only checking what is in the package metadata.

Luke, I guess that is an automatic thing. There is no dash in the metadata that explains the issue. Can you change how cc checks the name?

[daf]

Can you change how cc checks the name?

CC isn't doing anything here, it's just using normal python packaging (whatever that means).

[rsignell-usgs]

Or @marqh, any ideas here?

[pelson]

There is nothing in cf_units which uses cf-units as far as I can tell: https://github.com/SciTools/cf_units. Nor can I see anything in the conda recipe at https://github.com/SciTools/conda-recipes-scitools/blob/master/cf_units/meta.yaml.

I think I'm missing something...

[daf]

@pelson , I think @lukecampbell discovered it's an automated thing inside of setuptools that does the substitution for you (how user-friendly!).

[rsignell-usgs]

It certainly would be more standard to call this "cf-units" rather than "cf_units" anyway, no?
Or is it too late for that?

[ocefpaf]

It certainly would be more standard to call this "cf-units" rather than "cf_units" anyway, no?

Or is it too late for that?

I don't recall, but we had a reason to use the underscore.

[lukecampbell]

Just fair warning there is another library that's not quite as good called "cfunits". That caused us some confusion and problems here.

[rsignell-usgs]

Yep, we know about that package. Hence the need to call this package something like "cf-units".

I suppose we could also call it "cf-units-use-this-one" :smile_cat:

[daf]

Speaking with @kwilcox and confirmed by a quick test, conda install -c ioos cf_units installed BOTH of these:

cf-units                  1.1                       <pip>
cf_units                  1.1                      py34_0    ioos

despite only listing it once in the "things I will install".

[marqh]

I confess I'm quite puzzled now. It appears that cf_units 1.1 has not been released to pypi https://pypi.python.org/pypi?%3Aaction=search&term=cf_units&submit=search which is an omission, and ought to be fixed. cf-units doesn't exist in pypi

So, I don't see how a cf_units 1.1 could be sourced from pip

similarly to @pelson, I cannot see where the hyphen name (cf-units) comes from

[lukecampbell]

cf_units 1.1 comes from the conda install here https://github.com/SciTools/conda-recipes-scitools/tree/40cefa5034c34e453e811dc1adcd70791bf04874/cf_units I'm not sure why it says pip, maybe the egg is compiled by conda and then pip installed as an egg from a conda repo?

the hyphen is a convention used by setuptools. They will replace any underscore in a package's name with a hyphen.

http://pybuilder.github.io/documentation/external_plugins.html#.VsX1U5MrLao

[marqh]

the hyphen is a convention used by setuptools. They will replace any underscore in a package's name with a hyphen.

i did not know this, that is quite a confusing thing to do, imho

[rsignell-usgs]

So can we move forward by calling this package cf-units instead of cf_units. That would make life a lot easier, no?

[lukecampbell]

Does anyone on this thread have the privilege to make that happen?

[ocefpaf]

Does anyone on this thread have the privilege to make that happen?

Again, I don't remember why, but we had reasons to use the underscore.

I don't think it us necessary to rename the package. As soon as I get back I'll take a look at cc and check why (and how) it is using setuptools PKG-INFO to avoid that. And/or see if it is possible to make setuptools use the correct name. (I haven't read yet the reasons for that behavior.)

[kknee]

Folks and @jbosch-noaa specifically: Version 2.1 has some important updates. Its release is being held up by this issue. Thoughts on punting this to a 2.1.1 release so that we can get the other updates out?

[daf]

I guess it's fine as this mistake was already included in the 2.0 release and is therefore broken. Basically, you must use conda to use compliance-checker until this is resolved.

Alternately, we can revert cf-units dep back to 0.1.0 as a temporary measure, not sure what that would entail, @ocefpaf ?

EDIT: i can't read

[lukecampbell]

You mean for windows only right? I can use compliance checker as a pip install no prob.

pip install numpy netCDF4
pip install git+https://github.com/ioos/compliance-checker.git\#egg=compliance-checker
Collecting compliance-checker from git+https://github.com/ioos/compliance-checker.git#egg=compliance-checker
  Cloning https://github.com/ioos/compliance-checker.git to /private/var/folders/q_/ycb36mqx74dcfxll32g_8xm42f_vz7/T/pip-build-_D97Yz/compliance-checker
Collecting OWSLib>=0.8.3 (from compliance-checker)
  Using cached OWSLib-0.10.3.tar.gz
Collecting lxml>=3.2.1 (from compliance-checker)
Collecting cf-units>=0.1.1 (from compliance-checker)
  Using cached cf_units-1.0.0.tar.gz
Collecting requests>=2.2.1 (from compliance-checker)
  Using cached requests-2.9.1-py2.py3-none-any.whl
Collecting python-dateutil>=2.2 (from compliance-checker)
  Using cached python_dateutil-2.4.2-py2.py3-none-any.whl
Collecting Jinja2>=2.7.3 (from compliance-checker)
  Using cached Jinja2-2.8-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): setuptools>=15.0 in /Users/lcampbell/.venvburrito/lib/python2.7/site-packages/setuptools-18.2-py2.7.egg (from compliance-checker)
Collecting pytz (from OWSLib>=0.8.3->compliance-checker)
  Using cached pytz-2015.7-py2.py3-none-any.whl
Collecting pyproj (from OWSLib>=0.8.3->compliance-checker)
  Using cached pyproj-1.9.5.1.tar.gz
Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in /Users/lcampbell/.venvburrito/lib/python2.7/site-packages (from python-dateutil>=2.2->compliance-checker)
Collecting MarkupSafe (from Jinja2>=2.7.3->compliance-checker)
Building wheels for collected packages: OWSLib, cf-units, pyproj
  Running setup.py bdist_wheel for OWSLib
  Stored in directory: /Users/lcampbell/Library/Caches/pip/wheels/6b/1a/4c/b3bc8fc46b1b009a8e800fcd8c8ddd03dd82bdb3467d355b42
  Running setup.py bdist_wheel for cf-units
  Stored in directory: /Users/lcampbell/Library/Caches/pip/wheels/52/93/a7/8463e80e35a37cd2a4d0d25cdaadd4a9f314d978bf3045e035
  Running setup.py bdist_wheel for pyproj
  Stored in directory: /Users/lcampbell/Library/Caches/pip/wheels/19/5c/94/930f385a4fb2a30f3c1d64280777cb596c4322c3dff4ef1ece
Successfully built OWSLib cf-units pyproj
Installing collected packages: python-dateutil, pytz, requests, pyproj, OWSLib, lxml, cf-units, MarkupSafe, Jinja2, compliance-checker
  Running setup.py install for compliance-checker
Successfully installed Jinja2-2.8 MarkupSafe-0.23 OWSLib-0.10.3 cf-units-1.0.0 compliance-checker-2.0.0 lxml-3.5.0 pyproj-1.9.5.1 python-dateutil-2.4.2 pytz-2015.7 requests-2.9.1

(test)➜  oceansmap git:(master) compliance-checker -t acdd -c strict http://data.ioos.us/gliders//thredds/dodsC/deployments/mkhoward/bass-20150706T151619Z/bass-20150706T151619Z.nc3.nc

Running Compliance Checker on the dataset from: http://data.ioos.us/gliders//thredds/dodsC/deployments/mkhoward/bass-20150706T151619Z/bass-20150706T151619Z.nc3.nc

--------------------------------------------------------------------------------
                    The dataset scored 127 out of 152 points
                             during the acdd check
--------------------------------------------------------------------------------

[daf]

~~Collecting cf-units>=0.1.1 (from compliance-checker) Using cached cf_units-1.0.0.tar.gz~~

EDIT: i can't even read

[jbosch-noaa]

OMG! How confusing!! @kknee -If this issue will take longer than the rest of this week to solve then I say release V2.1 and down the line, once "-" and " _ " syntax are resolved, release a 2.1.1

[kknee]

@ocefpaf - you were the latest one to volunteer looking into this. Do you see a fix happening this week? If not, and barring any nays on a release without this fix, we'll get it out this week.

[jbosch-noaa]

@ocefpaf is a Ocean Sciences this week so I doubt he will get to it.

[daf]

I appear to be having a day of version number dyslexia. Reading 1.0.0, seeing 0.1.0, sorry for the distraction.

Let's get back to Rich's main issue, which is Windows related.

[ocefpaf]

@ocefpaf - you were the latest one to volunteer looking into this. Do you see a fix happening this week? If not, and barring any nays on a release without this fix, we'll get it out this week.

I have a few ideas, but I won't be able to test them until I get back home. (Next Tuesday.)

[kknee]

@ocefpaf have you had a chance to put any more thought into this? Would be nice to include in the next release.

[ocefpaf]

@ocefpaf have you had a chance to put any more thought into this? Would be nice to include in the next release.

Nope. I forgot about this. (My is buried into conda-forge for the moment.)

I will look into this before this weekend.

[jbosch-noaa]

@ocefpaf - It's almost been a month and I never heard any follow-up on this. Are we good with it yet?

[ocefpaf]

@ocefpaf - It's almost been a month and I never heard any follow-up on this. Are we good with it yet?

I am re-tracing the issue here locally to see if I can reproduce it. I will have an answer later today. Hold on...

[ocefpaf]

I am using the conda-forge channel (which supersedes the ioos channel).

conda create --name TEST --channel conda-forge python=3.4 compliance-checker
source activate TEST
conda list | grep 'cf[-_]units'
cf-units                  1.1                       <pip>
cf_units                  1.1                      py34_0    conda-forge

That show "both" versions, but it is actually just one. I guess that conda is misinterpreting the fact that setuptools changes the _ to - in the name metadata. Note that this should not hurt compliance-checker functionality. I still can:

url="http://geoport.whoi.edu/thredds/dodsC/clay/usgs/users/jcwarner/Projects/Sandy/triple_nest/00_dir_NYB05.ncml"
compliance-checker --test cf $url

Running Compliance Checker on the dataset from: http://geoport.whoi.edu/thredds/dodsC/clay/usgs/users/jcwarner/Projects/Sandy/triple_nest/00_dir_NYB05.ncml

--------------------------------------------------------------------------------
                    The dataset scored 881 out of 931 points                    
                              during the cf check                               
--------------------------------------------------------------------------------
                               Scoring Breakdown:                               

all_features_are_same_type              :2:     0/0
contiguous_ragged_array                 :2:     0/0
coordinates_and_metadata                :2:     0/0
feature_type                            :2:     0/0
incomplete_multidim_array               :2:     0/0
indexed_ragged_array                    :2:     0/0
missing_data                            :2:     0/0
orthogonal_multidim_array               :2:     0/0

<truncate output>

I test this on Linux (py27, py34, and py35) and Windows-32 (py27 and py34). They all worked. (Though I did find a unicode bug on Windows when using Python 3.4 I will report that later at compliance-checker issue tracker.)

Rich I don't have a Windows-64 machine, but can you test that again to see if you can reproduce the original error?

I am inclined to say that this is just a bug on how conda reports the package metadata. No renaming should be necessary.

[rsignell-usgs]

Well, we don't seem to have that problem anymore on win64, but we do seem to have another problem:

[TEST] C:\Users\rsignell>conda create --name TEST --channel conda-forge python=3.4 compliance-checker

[TEST] C:\Users\rsignell>compliance-checker --test cf http://geoport.whoi.edu/thredds/dodsC/clay/usgs/users/jcwarner/Projects/Sandy/triple_nest/00_dir_NYB05.ncml

...
indexed_ragged_array                    :2:     0/0
missing_data                            :2:     0/0
orthogonal_multidim_array               :2:     0/0
                                 High Priority

--------------------------------------------------------------------------------

    Name                            :Priority: Score
Traceback (most recent call last):
  File "C:\Users\rsignell\AppData\Local\Continuum\Miniconda2\envs\TEST\Scripts\c
ompliance-checker-script.py", line 5, in <module>
    sys.exit(main())
  File "C:\Users\rsignell\AppData\Local\Continuum\Miniconda2\envs\TEST\Scripts\c
checker.py", line 41, in main
    args.format)
  File "C:\Users\rsignell\AppData\Local\Continuum\Miniconda2\envs\TEST\lib\site-
packages\compliance_checker\runner.py", line 44, in run_checker
    groups = cls.stdout_output(cs, score_groups, verbose, limit)
  File "C:\Users\rsignell\AppData\Local\Continuum\Miniconda2\envs\TEST\lib\site-
packages\compliance_checker\runner.py", line 74, in stdout_output
    cs.non_verbose_output_generation(score_list, groups, limit, points, out_of)
  File "C:\Users\rsignell\AppData\Local\Continuum\Miniconda2\envs\TEST\lib\site-
packages\compliance_checker\suite.py", line 311, in non_verbose_output_generatio
n
    print('%-40s:%s:%6s/%1s'  % (score_list[x][0][0:39], score_list[x][1], score
_list[x][2][0], score_list[x][2][1]))
  File "C:\Users\rsignell\AppData\Local\Continuum\Miniconda2\envs\TEST\lib\encod
ings\cp437.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\xa7' in position 0:
 character maps to <undefined>

[TEST] C:\Users\rsignell>