import_reg command failing due to renamed import

[gregoryfoster]

Hello, I'm new-ish to the project and was attempting to import the local output from the suggested regulations-parser example regulation (Title 27 Part 447) into my instance of regulations-core using the suggested import_reg command. I saw that Python 2.7 is required for that command due to its usage of urlparse, which may be worth documenting as a separate issue if/for when the project should move to depend only on Python 3. The command then presented the following error:

File "...regulations-core/regcore/management/commands/import_reg.py", line 13, in <module>
    from regcore_write.views import regulation, diff, layer, notice
ImportError: cannot import name regulation

Seeing there's no longer a regulation.py file in regcore_write.views I was able to identify a commit (2a5f8cdbd38a7899c4f210a103bb4f598afc8520) which shows the object was renamed to Document. Changing the import plus the only reference I could see (line #201) resulted in the command running successfully and some rows added to the SQLite database - but no rows were added to regcore_document which seems suspicious to me. There are quite a few other variables in the command which reference "regulation" and perhaps some expectation of values named accordingly in the JSON files, so I wanted to check with the experts before submitting a minimal pull request.

Thank you for your important work on this project, more timely than ever!

[cmc333333]

@gregoryfoster eep! That command needs a pretty big overhaul. For the most part, we directly write from the parser to a running API (via eregs pipeline 11 22 http://running-api/).

While we update that file, try this: run the API (e.g. via python manage.py runserver), and use curl to POST all of the documents:

cd /path/to/dir-that-contains-notice-regulation-etc/
for TAIL in $(find */* -type f | sort -r)
do
    curl -X PUT http://localhost:8000/$TAIL -d @$TAIL
done

[gregoryfoster]

Hi @cmc333333! I'd be happy to help on this if you can tell me a little about the refactoring from regulation to document. I'm looking to learn much more about the internals of the eregs code bases, but I understand if you would recommend against this being a good place to jump in.

And thanks for the script to POST the generated documents to the running server, it's nice to witness your bash skillz :) I'd still like to get this manual workflow operational as I like reviewing the JSON output with the certainty that comes with one less potential point of failure. All good though, and not a high priority.

[cmc333333]

@gregoryfoster we'd be happy to have you aboard! If you have a moment, please check out PR #66 to see if it works for you. In this case, the module was more of a technical debt issue than a learn-about-the-codebase issue :) If you're interested in diving in, we'll soon want to:

• migrate our tests to py.test style
• verify and cleanup any issues around Django 1.10
• add the linting plugins to match -parser
• (potentially) remove the Elastic-as-database option (which would greatly simplify this app)

I'd be happy to guide you through any of that effort if you have time. That said, -core is probably the least interesting of the three main repos. It was originally a quick Flask + Elastic Search app to act as a shared datastore. Though it's now (generally) backed by RDS and runs Django, it's not moved too far from those data-in/data-out origins. -parser does most of the data processing (indeed, a longer-term goal is extend -parser with -core's functionality) and -site converts that data into a usable interface; both tend to grab mind-share more than this repo.

[gregoryfoster]

PR #66 looks good to me and I appreciate the simplified syntax. I've now got documents in regcore_document. Thanks, @cmc333333!