ARXIVNG-1440 user can compile their submission and preview it

[erickpeirson]

Ok kids, here's the compiler integration. These changes depend on changes in a few other places:

• https://github.com/arXiv/arxiv-submission-core/pull/53
• https://github.com/arXiv/arxiv-filemanager/pull/32 (I already built and pushed this as arxiv/filemanager:0.1rc2 so that you can use it)
• https://github.com/arXiv/arxiv-compiler/pull/12

To run the compiler you need the underlying converter image. You will need to download this image, which is enormous. So get yourself tapped into the mainline, or whatever.

docker pull [ will share via slack ]:0.8

To really see this in action, you'll want to run this with all of the backend services (including the submission worker). The easiest way to do this is to use the docker-compose config included in the root of this repository.

Make sure that you have the latest version of docker-compose.

cd /path/to/arxiv-submission-ui
mkdir /tmp/foo     # Compiler service will use this.
HOST_SOURCE_ROOT=/tmp/foo COMPILER_DOCKER_IMAGE=[ will share via slack ]:0.8 docker-compose build
HOST_SOURCE_ROOT=/tmp/foo COMPILER_DOCKER_IMAGE=[ will share via slack ]:0.8 docker-compose up

Note! This will start a whole bunch of stuff. Fairly late in the process, a bootstrap process will run and generate a bunch of users who are authorized to submit things. It will look something like this:

submission-bootstrap     | 1 picoline2058@gmail.com eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzZXNzaW9uX2lkIjoiYWQyY2UxNmUtMjQwNi00NjgwLWI2NWItMDE3NGYyNDA0MzhlIiwic3RhcnRfdGltZSI6IjIwMTktMDItMTNUMTI6MjM6MzQuMjE2NzcxLTA1OjAwIiwidXNlciI6eyJ1c2VybmFtZSI6InBpY29saW5lMjA1OEBnbWFpbC5jb20iLCJlbWFpbCI6InBpY29saW5lMjA1OEBnbWFpbC5jb20iLCJ1c2VyX2lkIjoxLCJuYW1lIjp7ImZvcmVuYW1lIjoiTWF1cmEiLCJzdXJuYW1lIjoiWmFyZW1iYSIsInN1ZmZpeCI6IlBhbiJ9LCJwcm9maWxlIjp7ImFmZmlsaWF0aW9uIjoiQ29ybmVsbCBVbml2ZXJzaXR5IiwiY291bnRyeSI6InVzIiwicmFuayI6Mywic3VibWlzc2lvbl9ncm91cHMiOlsiZ3JwX3BoeXNpY3MiXSwiZGVmYXVsdF9jYXRlZ29yeSI6ImFzdHJvLXBoLkdBIiwiaG9tZXBhZ2VfdXJsIjoiIiwicmVtZW1iZXJfbWUiOnRydWV9LCJ2ZXJpZmllZCI6ZmFsc2V9LCJjbGllbnQiOm51bGwsImVuZF90aW1lIjoiMjAxOS0wMi0xM1QyMjoyMzozNC4yMTY3NzEtMDU6MDAiLCJhdXRob3JpemF0aW9ucyI6eyJjbGFzc2ljIjowLCJlbmRvcnNlbWVudHMiOlsiKi4qIl0sInNjb3BlcyI6WyJwdWJsaWM6cmVhZCIsInN1Ym1pc3Npb246Y3JlYXRlIiwic3VibWlzc2lvbjp1cGRhdGUiLCJzdWJtaXNzaW9uOnJlYWQiLCJ1cGxvYWQ6cmVhZCIsInVwbG9hZDp1cGRhdGUiLCJ1cGxvYWQ6ZGVsZXRlIiwidXBsb2FkOnJlYWRfbG9ncyJdfSwiaXBfYWRkcmVzcyI6bnVsbCwicmVtb3RlX2hvc3QiOm51bGwsIm5vbmNlIjpudWxsfQ.iNOiCGVIZi5iipElLRyUlnx9uucdK7aytjkvr87FTvI
submission-bootstrap     | 2 aggregat2070@gmail.com eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzZXNzaW9uX2lkIjoiODhmYzMwYTUtNWMyMi00N2ZlLWIzMzEtMDJkNWFhNjUxZjRkIiwic3RhcnRfdGltZSI6IjIwMTktMDItMTNUMTI6MjM6MzQuMjIxNDQ3LTA1OjAwIiwidXNlciI6eyJ1c2VybmFtZSI6ImFnZ3JlZ2F0MjA3MEBnbWFpbC5jb20iLCJlbWFpbCI6ImFnZ3JlZ2F0MjA3MEBnbWFpbC5jb20iLCJ1c2VyX2lkIjoyLCJuYW1lIjp7ImZvcmVuYW1lIjoiQmlsZ2UiLCJzdXJuYW1lIjoiT2t1bXVcdTAxNWYiLCJzdWZmaXgiOiJQcm9mLiJ9LCJwcm9maWxlIjp7ImFmZmlsaWF0aW9uIjoiQ29ybmVsbCBVbml2ZXJzaXR5IiwiY291bnRyeSI6InVzIiwicmFuayI6Mywic3VibWlzc2lvbl9ncm91cHMiOlsiZ3JwX3BoeXNpY3MiXSwiZGVmYXVsdF9jYXRlZ29yeSI6ImFzdHJvLXBoLkdBIiwiaG9tZXBhZ2VfdXJsIjoiIiwicmVtZW1iZXJfbWUiOnRydWV9LCJ2ZXJpZmllZCI6ZmFsc2V9LCJjbGllbnQiOm51bGwsImVuZF90aW1lIjoiMjAxOS0wMi0xM1QyMjoyMzozNC4yMjE0NDctMDU6MDAiLCJhdXRob3JpemF0aW9ucyI6eyJjbGFzc2ljIjowLCJlbmRvcnNlbWVudHMiOlsiKi4qIl0sInNjb3BlcyI6WyJwdWJsaWM6cmVhZCIsInN1Ym1pc3Npb246Y3JlYXRlIiwic3VibWlzc2lvbjp1cGRhdGUiLCJzdWJtaXNzaW9uOnJlYWQiLCJ1cGxvYWQ6cmVhZCIsInVwbG9hZDp1cGRhdGUiLCJ1cGxvYWQ6ZGVsZXRlIiwidXBsb2FkOnJlYWRfbG9ncyJdfSwiaXBfYWRkcmVzcyI6bnVsbCwicmVtb3RlX2hvc3QiOm51bGwsIm5vbmNlIjpudWxsfQ.dZdha4zYX9KYsCCucCxcTNVFQQdV4p-ml00XvKKI2zY

Those are some JWTs you can use to access the submission UI. You will need to pass the JWT in the Authorization header.

Compile!

Ok, here's what it should look like when you arrive at the process step. I hid some things that weren't relevant for the first version (e.g. choosing preferred compiler).

Here's what it looks like when compilation starts:

Here's what it looks like when processing is successful:

Here's what it looks like when processing fails:

Known issues/not implemented

Unknown submission type

"Unknown submission type" is displayed when initially uploading things (even adding files later). This should be an easy fix.

It goes away if you reload the page:

Handling based on file type

This is next! Now that we have the submission type, we can figure out what to do about compilation (or not). This needs to be implemented.

Better handling of connection errors, other oddities

Stuff happens in the wild. Even moving from Flask to uWSGI surfaced a bunch of little annoyances. Need a fresh pass for exception handling around service-to-service communication.

I'm sure that there are other things

Please break it!

[erickpeirson]

And I see that pylint is failing, so there's that, too

[erickpeirson]

Oh and the page doesn't reload on its own yet.

[erickpeirson]

And you should all be aware of this for the future: https://github.com/unbit/uwsgi/issues/1126

[eawoods]

the command docker pull [ image as given ] produces 'Invalid reference format` as the only output when run from CLI within arxiv-compiler. Details I'm missing?

[eawoods]

Setting a (successfully compiled) submission to 'published' and then clicking back to replacement: Process step is highlighted already in the progress bar for v2 (@erickpeirson would you rather have this type of feedback as a ticket?)

[erickpeirson]

@eawoods I suspect we'll have quite a few little oddities to stomp. Let's make issues out of them, and then circle back?

[erickpeirson]

@JaimieMurdock @DavidLFielding Planning to merge this by COB Monday unless you see major problems

[DavidLFielding]

I'm able to generate the hung "Processing underway" state that Liz reported earlier.

[erickpeirson]

I'm able to generate the hung "Processing underway" state that Liz reported earlier.

It's always nice when we can break things reproducibly... ;-)

[erickpeirson]

Ok; going to merge this. There is still a lot of massaging to do, but the basic ability to compile and preview is here, and we can chip away in more bite sized chunks.