Django issues

[taldcroft]

Now the fun begins.

Django is a package that can't be allowed to float even in an uncontrolled bleeding edge Ska3. Django 2.x definitely does not work. I tried asking conda to downgrade to 1.6 but it wasn't happy:

(ska3) neptune$ conda install django=1.6
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - django=1.6
  - xija
Use "conda info <package>" to see the dependencies for each package.

Conda would accept django 1.8, so I tried that. That runs tests with some failures, though I'm not sure if they are due to using 1.8 or due to my standalone mac environment.

So I force uninstalled django 1.8 and installed django 1.6.1 from the tarball. That was worse:

In [1]: import kadi

In [2]: kadi.test()
=========================================================== test session starts ===========================================================
platform darwin -- Python 3.6.5, pytest-3.6.2, py-1.5.4, pluggy-0.6.0
rootdir: /Users/aldcroft/miniconda3/envs/ska3/lib/python3.6/site-packages, inifile:
plugins: remotedata-0.3.0, openfiles-0.2.0, doctestplus-0.1.2, arraydiff-0.2
collected 35 items / 1 errors                                                                                                             

================================================================= ERRORS ==================================================================
_______________________________________________ ERROR collecting kadi/tests/test_events.py ________________________________________________
kadi/tests/test_events.py:6: in <module>
    from .. import events
kadi/events/__init__.py:59: in <module>
    from .query import *
kadi/events/query.py:11: in <module>
    from . import models
kadi/events/models.py:9: in <module>
    from django.db import models
django/db/models/__init__.py:5: in <module>
    from django.db.models.query import Q
django/db/models/query.py:14: in <module>
    from django.db.models.fields import AutoField
django/db/models/fields/__init__.py:15: in <module>
    from django import forms
django/forms/__init__.py:8: in <module>
    from django.forms.fields import *
django/forms/fields.py:17: in <module>
    from django.forms.util import ErrorList, from_current_timezone, to_current_timezone
django/forms/util.py:4: in <module>
    from django.utils.html import format_html, format_html_join
django/utils/html.py:14: in <module>
    from .html_parser import HTMLParser, HTMLParseError
django/utils/html_parser.py:12: in <module>
    HTMLParseError = _html_parser.HTMLParseError
E   AttributeError: module 'html.parser' has no attribute 'HTMLParseError'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================= 1 error in 1.36 seconds =========================================================

Of course kadi events was working before in my mac standalone Ska3, but just yesterday I did some housecleaning and blew it away.

[jeanconn]

Is this a case where the tests are sufficient? I mean, can we run whichever version of Django if I get the tests to pass?

[taldcroft]

Yes for ska3, tests are reasonably complete. The web server is a different story.

[taldcroft]

Slightly OT but my thought was basically pushing the versions of everything to the max for the initial release of ska-conda. There is Python 3.7 of course but that can take a little while to get supported on conda.

[jeanconn]

I think I understand you. I was working from somewhat the other direction, as I was basically building in our current py3 (with a few pinned packages) and and was planning to try to move things more toward the max after I got those tests to pass.

[jeanconn]

It looks like this django version is working in my test/build linux environment just now

    django:             1.11.3-py36_0         defaults                                          

I think there is no django 1.8 build on py 3.6 in the default channel. We could also build either 1.6 or 1.8 or some other version custom.

Looks like my tests are OK except for fails on lucky

rootdir:/proj/sot/ska/jeanproj/git/ska-conda, inifile:
collected 46 items                                                                                                         

kadi/commands/tests/test_commands.py ...
kadi/commands/tests/test_states.py .................x.........
kadi/tests/test_events.py ...........
kadi/tests/test_occweb.py .F...

[taldcroft]

Cool. Mac?

[jeanconn]

It looks like remaining problems with the tests on Mac are not due to django. Some kadi cmd_states regression match errors mostly (and it wanted an eng_archive). Also had to copy events.db3 cmds.pkl cmd_states.h5 and such to /proj/sot/ska/data areas on my laptop. Not sure what is set for ska_path type stuff or which tests should skip if there's no local data.

platform darwin -- Python 3.6.2, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /proj/mconda3/envs/ktest/lib/python3.6/site-packages, inifile:
collected 46 items

kadi/commands/tests/test_commands.py ...
kadi/commands/tests/test_states.py ..FF.F......F....x.........
kadi/tests/test_events.py ...........
kadi/tests/test_occweb.py sssss

[jeanconn]

I have not figured out repo management at all yet though (do we move the files in ska3-conda and start again or add to it)?

[jeanconn]

I deleted the ska3-conda files and started again with these new builds.

[taldcroft]

Seems like the ska package is missing for OS-64:

neptune$ conda search ska
Loading channels: done

PackagesNotFoundError: The following packages are not available from current channels:

  - ska

Current channels:

  - https://conda.anaconda.org/sherpa/osx-64
  - https://conda.anaconda.org/sherpa/noarch
  - https://---@cxc.cfa.harvard.edu/mta/ASPECT/ska3-conda/osx-64
  - https://---@cxc.cfa.harvard.edu/mta/ASPECT/ska3-conda/noarch
...

[taldcroft]

About package management, I think in general we just add to the packages, just like with $ska/pkgs/. At some point we might do cleanup but that wouldn't be an immediate concern.

[jeanconn]

Yes, from the docs I thought you wanted it called ska-flight so I renamed while reorganizing. (it obviously needs more work to actually be flight and not sure which repo it will end up in).

[jeanconn]

I cleaned up this time around because I really don't know which package it picks if you have more than one with the same version (which is not a problem we have in $ska/pkgs at present).

[taldcroft]

OK, in #7 I saw ska at the end.

[jeanconn]

Fair enough, this time through I didn't use the build order so didn't catch that.