search

PreTeXtBook / pretext

PreTeXt: an authoring and publishing system for scholarly documents
https://pretextbook.org
Other
254 stars 203 forks source link

initial progress toward local pg processing #2014

Open drdrew42 opened 10 months ago

drdrew42 commented 10 months ago

Create webwork-representations locally

This is still a work in progress. I am sharing it as a draft PR for visibility and feedback.

Prerequisites

Local install of Standalone PG Renderer

Place the attached ptx script into script/ folder of the renderer. Rename it to ptx and make sure it is executable...

Copy any WeBWorK macro dependencies (e.g. WWSC.pl) into the private/macros/ folder of the renderer.

Usage

Run pretext/pretext -c webwork --pgscript /absolute-path/to-renderer/script/ptx <other config + project files>

Status

Line 1100 of pretext.py requires an internet connection in order to sanitize_url, I did bad things™️ in order to support fully offline use. This still needs to be resolved.

Next Steps

Update publisher file to support problem rendering by the standalone renderer instead of webwork2

sean-fitzpatrick commented 10 months ago

Looks like there is a typo in the usage instructions: --pgScript should be --pgscript.

I'm running this now against APEX. Update pending in 30 minutes or so.

sean-fitzpatrick commented 10 months ago

OK... container is up and running on the mojo-templates branch. Looks like the docker build missed a Perl module. After successful application of extract-pg.xsl I get the following error:

Can't locate Mojo/DOM.pm in @INC (you may need to install the Mojo::DOM module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.36.0 /usr/local/share/perl/5.36.0 /usr/lib/x86_64-linux-gnu/perl5/5.36 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.36 /usr/share/perl/5.36 /usr/local/lib/site_perl) at /home/sean/webwork-renderer/container/script/ptx line 8.
BEGIN failed--compilation aborted at /home/sean/webwork-renderer/container/script/ptx line 8.
Traceback (most recent call last):
  File "/home/sean/Documents/pretext/pretext/pretext", line 819, in <module>
    main()
  File "/home/sean/Documents/pretext/pretext/pretext", line 663, in main
    ptx.webwork_to_xml(
  File "/home/sean/Documents/pretext/pretext/pretext.py", line 1581, in webwork_to_xml
    response_root = ET.fromstring(response_text)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/lxml/etree.pyx", line 3257, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 1916, in lxml.etree._parseMemoryDocument
  File "src/lxml/parser.pxi", line 1796, in lxml.etree._parseDoc
  File "src/lxml/parser.pxi", line 1085, in lxml.etree._BaseParser._parseUnicodeDoc
  File "src/lxml/parser.pxi", line 618, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 728, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 657, in lxml.etree._raiseParseError
  File "<string>", line 1
lxml.etree.XMLSyntaxError: Document is empty, line 1, column 1
drdrew42 commented 10 months ago

Mojo::DOM is part of the Mojolicious package, which is covered in the Dockerfile. Are you trying to run this against the container, or do you have a local install? (Docker installs the perl modules into the container and not onto your local machine...)

sean-fitzpatrick commented 10 months ago

I ran the docker run -d command from your repo with an added line --mount type=bind,source=/home/sean/webwork-renderer/container/,target=/usr/app/private \ to include the macros file for APEX.

Then I checked out your pull request and ran the pretext/pretext command as above.

Everything is from a terminal; I didn't shell into the docker container or anything like that.

drdrew42 commented 10 months ago

Sorry for the confusion. (omg, I think I've been talking to LLMs too much!)

In order for the renderer to do its thing, it must be able to run locally on your machine -- the docker container is only accessible as a webservice, and this setup is built to avoid the overhead of making such requests.

The renderer dockerfile lists the dependencies, which will need to be installed locally. If you are running Ubuntu, these packages can be installed just as they are listed in the dockerfile. Otherwise, you will need CPANMinus (recommended, but CPAN will also work) to install the packages that start with "lib" and end with "-perl". For example, installing libclass-accessor-perl would be done like this: cpanm Class::Accessor. There are additional dependencies that are not packaged for Ubuntu, and they are listed on line 49 of the dockerfile. They can be installed by copying that RUN command (you may skip everything following the "&&").

sean-fitzpatrick commented 10 months ago

OK! Took a little longer than planned to get all the packages installed and then kids came home from school.

This time we got off to a good start -- first 6 problems built without a hitch. Then we ran into the first one using WeBWorK's TikZ support. This problem also has a table, and I'm seeing the warnings you mentioned (these are related to table alignment and styling that I think is supported in PreTeXt but not WeBWorK). Complaints about "bottom", "midrule", "align", and "valgin". But it's the image that breaks things.

Error message:

Traceback (most recent call last):
  File "/home/sean/Documents/pretext/pretext/pretext", line 819, in <module>
    main()
  File "/home/sean/Documents/pretext/pretext/pretext", line 663, in main
    ptx.webwork_to_xml(
  File "/home/sean/Documents/pretext/pretext/pretext.py", line 1498, in webwork_to_xml
    if ww_image_scheme:
       ^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'ww_image_scheme' where it is not associated with a value

Source code for the problem causing the error:

      <exercise label="ex-limit-intro-numerical-graphical-1">
          <webwork xml:id="ex-limit-intro-numerical-graphical-1">
            <pg-code>
              do {
                $a = list_random(-1,1);
                $b = non_zero_random(-5,5,1);
                $c = non_zero_random(-5,5,1);
                if ($envir{problemSeed} == 1){
                  ($a,$b,$c) = (1,3,-5);
                };
                $f = Formula("x^2 + $b x + $c")->reduce;
                $l = $f->eval(x => $a);
              } until($l != 0);
              @tabin = ($a - 0.1, $a - 0.01, $a - 0.001, $a + 0.001, $a + 0.01, $a + 0.1);
              @tabout = map{$f->eval(x => $_)} (@tabin);
            </pg-code>
            <statement>
              <p>
                <m>\lim\limits_{x\to <var name="$a"/>}\left(<var name="$f"/>\right)</m>
              </p>
              <instruction>
                If the limit does not exist, you may type <c>DNE</c>.
                If you need to enter <m>\infty</m>, you may type <c>inf</c>.
              </instruction>
              <p>
                <var name="$l" width="10"/>
              </p>
            </statement>
            <solution>
              <p>
                For a numerical approximation, make a table:
              </p>
              <tabular>
                <row bottom="medium">
                  <cell><m>x</m></cell>
                  <cell><m><var name="$f"/></m></cell>
                </row>
                <row>
                  <cell><m><var name="$tabin[0]"/></m></cell>
                  <cell><m><var name="$tabout[0]"/></m></cell>
                </row>
                <row>
                  <cell><m><var name="$tabin[1]"/></m></cell>
                  <cell><m><var name="$tabout[1]"/></m></cell>
                </row>
                <row>
                  <cell><m><var name="$tabin[2]"/></m></cell>
                  <cell><m><var name="$tabout[2]"/></m></cell>
                </row>
                <row>
                  <cell><m><var name="$tabin[3]"/></m></cell>
                  <cell><m><var name="$tabout[3]"/></m></cell>
                </row>
                <row>
                  <cell><m><var name="$tabin[4]"/></m></cell>
                  <cell><m><var name="$tabout[4]"/></m></cell>
                </row>
                <row>
                  <cell><m><var name="$tabin[5]"/></m></cell>
                  <cell><m><var name="$tabout[5]"/></m></cell>
                </row>
              </tabular>
              <p>
                For a graphical approximation:
              </p>
              <image>
                <latex-image>
                  \begin{tikzpicture}
                    \begin{axis}[grid=both]
                      \addplot+[domain=-2:2] {$f};
                    \end{axis}
                  \end{tikzpicture}
                </latex-image>
              </image>
              <p>
                It appears that when <m>x</m> is close to <m><var name="$a"/></m>,
                that <m><var name="$f"/></m> is close to <m><var name="$l"/></m>.
                So <m>\lim_{x\to <var name="$a"/>}\left(<var name="$f"/>\right)=<var name="$l"/></m>.
              </p>
            </solution>
          </webwork>
        </exercise>
rbeezer commented 10 months ago

The warnings about tables shouldn't matter here.

On June 27, 2023 9:52:22 PM PDT, Sean Fitzpatrick @.***> wrote:

OK! Took a little longer than planned to get all the packages installed and then kids came home from school.

This time we got off to a good start -- first 6 problems built without a hitch. Then we ran into the first one using WeBWorK's TikZ support. This problem also has a table, and I'm seeing the warnings you mentioned (these are related to table alignment and styling that I think is supported in PreTeXt but not WeBWorK). Complaints about "bottom", "midrule", "align", and "valgin". But it's the image that breaks things.

Error message:

Traceback (most recent call last):
 File "/home/sean/Documents/pretext/pretext/pretext", line 819, in <module>
   main()
 File "/home/sean/Documents/pretext/pretext/pretext", line 663, in main
   ptx.webwork_to_xml(
 File "/home/sean/Documents/pretext/pretext/pretext.py", line 1498, in webwork_to_xml
   if ww_image_scheme:
      ^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'ww_image_scheme' where it is not associated with a value

Source code for the problem causing the error:

     <exercise label="ex-limit-intro-numerical-graphical-1">
         <webwork xml:id="ex-limit-intro-numerical-graphical-1">
           <pg-code>
             do {
               $a = list_random(-1,1);
               $b = non_zero_random(-5,5,1);
               $c = non_zero_random(-5,5,1);
               if ($envir{problemSeed} == 1){
                 ($a,$b,$c) = (1,3,-5);
               };
               $f = Formula("x^2 + $b x + $c")->reduce;
               $l = $f->eval(x => $a);
             } until($l != 0);
             @tabin = ($a - 0.1, $a - 0.01, $a - 0.001, $a + 0.001, $a + 0.01, $a + 0.1);
             @tabout = map{$f->eval(x => $_)} ***@***.***);
           </pg-code>
           <statement>
             <p>
               <m>\lim\limits_{x\to <var name="$a"/>}\left(<var name="$f"/>\right)</m>
             </p>
             <instruction>
               If the limit does not exist, you may type <c>DNE</c>.
               If you need to enter <m>\infty</m>, you may type <c>inf</c>.
             </instruction>
             <p>
               <var name="$l" width="10"/>
             </p>
           </statement>
           <solution>
             <p>
               For a numerical approximation, make a table:
             </p>
             <tabular>
               <row bottom="medium">
                 <cell><m>x</m></cell>
                 <cell><m><var name="$f"/></m></cell>
               </row>
               <row>
                 <cell><m><var name="$tabin[0]"/></m></cell>
                 <cell><m><var name="$tabout[0]"/></m></cell>
               </row>
               <row>
                 <cell><m><var name="$tabin[1]"/></m></cell>
                 <cell><m><var name="$tabout[1]"/></m></cell>
               </row>
               <row>
                 <cell><m><var name="$tabin[2]"/></m></cell>
                 <cell><m><var name="$tabout[2]"/></m></cell>
               </row>
               <row>
                 <cell><m><var name="$tabin[3]"/></m></cell>
                 <cell><m><var name="$tabout[3]"/></m></cell>
               </row>
               <row>
                 <cell><m><var name="$tabin[4]"/></m></cell>
                 <cell><m><var name="$tabout[4]"/></m></cell>
               </row>
               <row>
                 <cell><m><var name="$tabin[5]"/></m></cell>
                 <cell><m><var name="$tabout[5]"/></m></cell>
               </row>
             </tabular>
             <p>
               For a graphical approximation:
             </p>
             <image>
               <latex-image>
                 \begin{tikzpicture}
                   \begin{axis}[grid=both]
                     \addplot+[domain=-2:2] {$f};
                   \end{axis}
                 \end{tikzpicture}
               </latex-image>
             </image>
             <p>
               It appears that when <m>x</m> is close to <m><var name="$a"/></m>,
               that <m><var name="$f"/></m> is close to <m><var name="$l"/></m>.
               So <m>\lim_{x\to <var name="$a"/>}\left(<var name="$f"/>\right)=<var name="$l"/></m>.
             </p>
           </solution>
         </webwork>
       </exercise>
sean-fitzpatrick commented 10 months ago

No, the table warnings aren't a concern. But the error when it hits the first TikZ image is.

rbeezer commented 10 months ago

Thanks, @drdrew42 for this, and putting it out early. I'll rely a lot on having @Alex-Jordan take a good hard look.

What if the PGscript path lived in the publisher file? Then use an existing method argument to pretext/pretext to be 'server' or 'standalone', which would be similar to how we get Asymptote images - from a local install or a service on a server at U of Alberta?

drdrew42 commented 10 months ago

Ultimately, I defer to how you want to implement-- I did this as proof of concept, and it was easiest to add a script parameter. Long term, my goal is to allow authors to select the renderer implementation for their activated WW problems, which makes sense in the publisher file.

There is a difference, though, between the two features. Compiling the output from pretext source using this script is independent of the choice to use "webwork2" vs. "renderer" for activated problems. The pgscript option is entirely dependent on the machine that compiles the pretext, while the (not yet written) option for activated problems in production only depends on the presence of a renderer service at a specified URL.

IMO, that makes pgscript less appropriate for a publisher file. Compiling a pretext book on another machine shouldn't require a change in the publisher file.

PS - I have only spotty internet for another couple weeks. I'll be back on this case once I return.

Alex-Jordan commented 2 months ago

@drdrew42 Looking at this now and seeing how far I get today. A year later, there is one tiny conflict that should be easy for you to resolve on your branch with a rebase.

Alex-Jordan commented 2 months ago

I've caught up to where you left off here :) I can build representations for the sample chapter using a local renderer in 14 seconds, where it takes 33 seconds when I use the AIM WW server.

Unless you object, I will start a new branch with your commits on it, clean some things up, and continue working toward rolling this out.

Alex-Jordan commented 2 months ago

Image files that the renderer produces are referenced in PG output as being at locations like:

standalonerenderer/tmp/images/e8e25e19-858b-3fd2-840a-9ff188f4e5d4___828928f1-23cf-3580-a0a1-68fcc3cf0f48.png

I need to replace that standalonerenderer with the actual path on my system to the renderer so that pretext can copy the image file to the generated/webwork/images folder. For example for me, I need to replace standalonerenderer with /Users/alex.jordan/renderer.

@rbeezer, if I have the standalone PG renderer installed somewhere on my system, where is an appropriate place for me to tell pretext where that is? A new place in pretext.cfg? A switch like the current -s for server location, but now for the path to the renderer?

Alternatively, @drdrew42 , can I get the renderer to just dump images directly into the generated/webwork/images folder?

And also for @drdrew42 , I have a more fundamental problem. The renderer does not seem to actually be creating image files and placing them in that tmp folder. Any idea what that could be about?

Aside from these issues with images, the representations file from using the renderer is close enough (for now) to the one I get from using the AIM WW server. The differences that I see are attributable to the AIM WW server using PG 2.17, and the rendererer is using something more recent.

Altern

rbeezer commented 2 months ago

A new place in pretext.cfg?

Yes, I think so - it sounds like another executable to me. A program of its own of to the side - not XSL, not even "called" by XSL.

But I'm hoping @drdrew42 has a better idea about flexibility for specifying this location.

Alex-Jordan commented 2 months ago

It's two things. One is an executable that Drew's usage (at top of this thread) prescribes via the --pgscript option. And then it's also got a tmp folder for work that it does. Which would be irrelevant to pretext except there is evidence the image files from processing land there, so pretext needs to get those.

Maybe the executable should move to pretext.cfg and --pgscript becomes an argument-free flag.

And I am hoping that somehow we can just tell the renderer a location to place image files in the first place, sidestepping the tmp folder issue.

drdrew42 commented 2 months ago

I'm on the road this weekend, but when I return, I will look into the possibility of what you've described.