Broken runestone directives in runestone > 5.0.8

[DaveParillo]

While attempting to upgrade an existing book to a later runestone version, noticed active code, parsons, fitb, tabbed, and other directives were not rendering. To take a possibly config migration / update out of the picture, I created a new book using runestone init. The default book does not render them either.

As an aside, the first code:: directive in section 4 is missing a blank line between the directive and the content. Should be:

.. code:: 

   html_static_path =  runestone_static_dirs() + ['_static']

The most recent release where I could all the sample init content to render was release 5.0.8.

Every other release (I only spot-checked 5.1.1, 5.2.1, 5.3.2, 5.3.0, and 5.5.0) generate books that do not render the example active code or multiple choice (or 5.2.1 for example renders just the code in a tiny text box and renders the multiple choice, but shows all the answers.

Runestone <= 5.0.8 have always worked in both debian and alpine.

Attempting to build books on a docker image based on python:slim:

FROM python:slim AS builder WORKDIR /usr/src/app COPY requirements.txt /tmp/

RUN mkdir -p /local/wheels \ && apt-get update \ && apt-get install -y \ build-essential \ gfortran \ wget \ libpng-dev \ libxml2-dev \ libxslt-dev \ && pip install --upgrade pip \ && pip install cython \ && pip wheel --wheel-dir=/local/wheels -r /tmp/requirements.txt

FROM python:slim AS runner RUN mkdir -p /local/wheels COPY requirements.txt /tmp/ COPY --from=builder /local/wheels /local/wheels RUN apt-get update \ && apt-get install -y \ graphviz \ vim \ && pip install --no-index --find-links=/local/wheels -r /tmp/requirements.txt \ && rm -r /local/wheels /tmp/requirements.txt \ && sed -i -e '/user not logged in/ d' /usr/local/lib/python3.9/site-packages/runestone/dist/runestone.js

VOLUME /var/book WORKDIR /var/book EXPOSE 8000 CMD [ "/bin/bash" ]

And requirements.txt:

runestone == 5.0.8 cython numpy scipy matplotlib

[bnmnetp]

Runestone.academy is using 5.5.4 with no problems.

We have unit tests for all of the components to make sure that everything renders properly and these tests are run as part of our Continuous Integration on Github. We would not make a release if the tests were failing.

Usually the quickest way to discover why one of the components is not rendering is to look in the Javascript console to see if there are any errors. But I'm wondering if your test book is even building correctly? I don't have any experience with python:slim but we use the python:3.8-buster image as the basis of our container in production.

[bnmnetp]

Also could you clarify which book you are referring to about the code directive? We have over 20 different books on runestone.academy I wouldn't even try to guess where to look.

[DaveParillo]

Brad, I'll try a 3.8-buster base image. The book is the one generated by "runestone init"

Dave

On Dec 29, 2020, at 1:08 PM, Bradley Miller notifications@github.com wrote:

Also could you clarify which book you are referring to about the code directive? We have over 20 different books on runestone.academy I wouldn't even try to guess where to look.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/RunestoneInteractive/RunestoneComponents/issues/1117#issuecomment-752241441, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACX6JVCEPERVK4PW53T6ZCTSXJAMPANCNFSM4VMYLBDA.

[bnmnetp]

Hmmm,

I wonder if it is your dynamic_pages setting. If you are building for a static site that should be False in pavement.py

[DaveParillo]

I always have set that to false, not really knowing what it is for, and it is the default for books created using 'runestone init'

Running in a docker image based on python:3.8-buster doesn't seem to change anything.

Something to note - if the content has already been build previously (say by using runestone 5.0.8) - everything refers fine. It's when I execute 'runestone build' that these problems arise.

When I build and run the 'init book', created using as many of the default values as possible, there are errors.

Parse error in runestone.js, line 5. After running through js-beautify --replace I can produce some useful line numbers. There are definitely compile issues in runestone.js:

1. a stray }(window, document) in the file creating a } mis-match on line 971.
2. The second to last line appears to contain a stray }]), which is both a typo and an extra function closing brace.
3. The last line looks like it should be };, not }));

But fixing these does not render the example directives. I don't know enough about sunstone inner workings to say if the issue is with the web pack process that creates runestone.js or one of the input files.

Probably unrelated, but noted: https://cdn.jsdelivr.net/npm/handsontable@7.2.2/dist/numbro.min.js.map returns a 404

-- Dave

On Dec 29, 2020, at 6:22 PM, Bradley Miller notifications@github.com wrote:

Hmmm,

I wonder if it is your dynamic_pages setting. If you are building for a static site that should be False in pavement.Ou

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[bnmnetp]

Well, if there is a parse error in runestone.js the explains why things are not rendering.

runestone.js is built from many .js files using webpack. I don't think it is meant to be run through js-beautify. In any case none of the versions that I have tried to pip install have syntax errors in runestone.js -- again the build process that creates our release would stop if it encountered problems bundling the js files.

But in an attempt to reproduce your issue if just tried creating several virtual environments and run the runestone init command to create a test book. 5.5.4 works just fine, 5.5.0 works just fine, 5.4.3 works just fine.

I do see the issue you referenced with the code block.

I'm trying to figure out what I can fix, but without the ability to reproduce the problem I'm stuck. I guess I can try to create a docker container using the information you gave me. But I don't see how that would be any different than creating a clean virtual environment.

[bnmnetp]

Here is the transcript of what I did. After creating a clean environment and running pip install runestone

 ◰³ testinit  ~/t/foo  runestone init                             829ms  Wed Dec 30 07:48:21 2020
This will create a new Runestone project in your current directory.
Do you want to proceed?  [Y/n]: y
Next we need to gather a few pieces of information to create your configuration files
Build book for dynamic page service? [False]:
Use Runestone Web Services  [false]:
Your Name  [bmiller]:
Title for this project  [Runestone Default]:
Use Simple Python3 Semantics  [false]:
Default ActiveCode language [python]:
Project name: (one word, no spaces):
Project name: (one word, no spaces): foobar
Path to build dir  [./build]:
Path to deploy built site  [../../static]:
Enable inline Activecode downloads by default (single activecode downloads may be enabled with the :enabledownload: flag) [false]:
Done. Type runestone build to build your project
 ◰³ testinit  ~/t/foo  runestone build --all serve --port 8000    32.3s  Wed Dec 30 07:48:55 2020
Building with Runestone 5.5.4
---> runestone.build
git describe --long
fatal: not a git repository (or any of the parent directories): .git

Building into  ./build/foobar
mkdir_p path('././build/foobar')
mkdir_p path('./build/foobar')
mkdir_p path('././build/foobar/doctrees')
sphinx-build -b html -d ././build/foobar/doctrees -c . -a -E -Alogin_required=false -Aloglevel=0 -Acourse_title=foobar -Apython3=false -Adburl=postgresql://bmiller:@localhost/bmiller -Adefault_ac_lang=python -Ajobe_server=http://jobe2.cosc.canterbury.ac.nz -Aproxy_uri_runs=/jobe/index.php/restapi/runs/ -Aproxy_uri_files=/jobe/index.php/restapi/files/ -Adownloads_enabled=false -Aenable_chatcodes=false -Aallow_pairs=false -Adynamic_pages=False -Ause_services=false -Abasecourse=foobar -Acourse_id=foobar -Aappname=runestone -Acourse_url=http://127.0.0.1:8000 -Arunestone_version=5.5.4 -Abuild_info=unknown ./_sources ./build/foobar
Running Sphinx v3.3.1
WARNING: while setting up extension runestone.lp: role 'docname' is already registered, it will be overridden
Connecting to DB
No module named 'psycopg2'
Skipping all DB operations because environment variables not set up
building [mo]: all of 0 po files
building [html]: all source files
updating environment: [new config] 1 added, 0 changed, 0 removed
Unable to save to source_code table in activecode.py. Possible problems:
  1. dburl or course_id are not set in conf.py for your book
  2. unable to connect to the database using dburl

This should only affect the grading interface. Everything else should be fine.

/Users/bmiller/tmp/foo/_sources/index.rst:89: WARNING: Error in "code" directive:
maximum 1 argument(s) allowed, 5 supplied.

.. code::
    html_static_path =  runestone_static_dirs() + ['_static']
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index
generating indices... genindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 2 warnings.

The HTML pages are in build/foobar.
Done, build successful
Note: this is a minimal static server without templates or a database.
For many use cases this is fine.
For the full server, see https://github.com/RunestoneInteractive/RunestoneServer
dynamic_pages is not defined
serving at port 8000

At this point all of the components in the sample textbook are rendering fine.

[DaveParillo]

Totally sympathize with not much you can do if you can't reproduce. I will try to nail it down tonight if I can. Your transcript looks identical to mine. The only real difference is I invoke runestone build and runestone serve as separate steps. I don't know the significance of runestone build --all, but doubt it's related.

My current half-baked theory du jour is that my multi-stage build is the issue. In the build stage I run pip wheel --wheel-dir=/local/wheels -r /tmp/requirements.txt, which include runestone. In the runner stage, I install the wheels built in the previous stage: pip install --no-index --find-links=/local/wheels -r /tmp/requirements.txt

This was more important when my docker image was alpine and I had to compile everything. I'm going to try to take the wheels completely out of the picture.

[DaveParillo]

Wheels are the problem. I don't know why runestone no longer likes to be turned into a wheel, but when I reverted to pip install everything works just fine. Not sure if you want to close this issue on the basis of this finding or change the title and figure out why runestone has an issue when built as a wheel.

[bnmnetp]

Hmm,

I guess this is another thing that I don't understand as well as I ought to. Here is what I do when I build a release for pypi.

python setup.py sdist
pip wheel --no-index --no-deps --global-option bdist_wheel  --wheel-dir dist dist/*.tar.gz
twine check dist/*
twine upload dist/*

And it is the wheel that is installed on Runestone Academy.

If you are just pip installing everything from PyPI why are you bothering with wheels at all? I feel like I'm missing something about your project.

[DaveParillo]

Looking that the documentation for pip wheel isn't shedding any light on why my wheel build was fundamentally different from yours, although I question whether I should have been using find-links during the build process. The issue may have been in how I was copying from my build layer to the final image layer.

As to "why" I was going through this for my book: My base docker image was alpine. Honestly, I didn't put much thought into that - it's a base I've used often, so it was a bit more of an autopilot thing. My book uses Matplotlib and scipy, which needed to be compiled from source. So I used a multi stage build for my docker machine, compiled wheels in the first stage, then copied the wheels and installed them in the final stage.

Installing on buster is far simpler - I'm just installing - nothing needs to be compiled. Much easier.

[bnmnetp]

Great. Is this your intermediate C++ book? Is it something you would be willing to have hosted on Runestone.academy for others to use?

[DaveParillo]

I have been using a version of it "cisc187-reader", partly because I thought the idea of me writing a "book" a bit pretentious and also because I wanted the flexibility to tailor the book exactly to my syllabus instead of trying to create a more generic second semester C++ text.

That said, I certainly have no problem having others share it, use it, or modify it.

It is licensed under the GFDL - not sure if that is a problem - I am unsure if it is compatible with the CC-BY-NC-SA license which I think everything in Runestone Academy uses. Originally, my intent was to try to guarantee as much freedom as possible, but I'm mostly interested in keeping it open and shareable.

On Jan 2, 2021, at 11:55 AM, Bradley Miller notifications@github.com wrote:

Great. Is this your intermediate C++ book? Is it something you would be willing to have hosted on Runestone.academy for others to use?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/RunestoneInteractive/RunestoneComponents/issues/1117#issuecomment-753522093, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACX6JVGDHRE3RGLN7HZLXCDSX522ZANCNFSM4VMYLBDA.

[bnmnetp]

Texts in Runestone use either GFDL or some variety of CC -- The python data structures book is specifically -NC- as Franklin Beedle generously allowed us to put the paper book into our online form and I would not want someone else to profit from that generosity.