Weave.jl analysis as a backup workflow

[jlperla]

With Weave.jl, we need to see if the following are possible (and if they are easy). Where it is not easily possible, we can discuss alternatives.

You can take over and replace my old attempts in https://github.com/ubcecon/computing_and_datascience/blob/master/julia_sandbox/Weave/README.md and https://github.com/ubcecon/computing_and_datascience/blob/master/julia_sandbox/Weave/

Basic Setup:

• [x] Setup a simple Weave.jl example there, focusing entirely on the .jmd input format.
• [ ] Modify instructions in https://github.com/ubcecon/computing_and_datascience/blob/master/julia_sandbox/Weave/README.md to your hearts content
• [x] Figure out output to (1) html, (2) PDF (3) jupyter notebook, (4) (tangled) .jl. All tests below would refer to all of these.
• [ ] Ultimately want to get the weave editor support working in Atom, but we can wait on that.

Things to check:

• [x] How do you do equation #s and references (including cross-referencing)
• [x] Can you split equations into multiple lines, as required.
• [x] How do you capture a Plots.jl and GR based output in all of the above output forms (except the tangling)
• [x] How to resize the image generated in the plot?
• [x] Figure labels and cross-references
• [x] How to do setup code which doesn't appear in the lecture notes, but would be geenrated in the .jl and .pynb output?
• [ ] How do you put in a HTML and PDF header/footer. Looks like we could modify https://github.com/mpastell/Weave.jl/tree/master/templates and run them.

Some of the things that I am pretty sure are not there, but we can discuss with others about:

• Any sense of bibtex references? I think we have a pandoc dependency here, so maybe we can hack something
• Any sort of testing approaches? Can we do a jldoctest style addition to Weave.jl? Can we use the tangled output for a test?

[arnavs]

Sounds good @jlperla. Thanks for breaking this down for me.

[jlperla]

Also note this PR: https://github.com/mpastell/Weave.jl/pull/126 which mentions https://github.com/Evizero/Augmentor.jl/blob/master/docs/exampleweaver.jl

[arnavs]

OK, @jlperla, I've spent some time with this and think this is done. Long story short, essentially all of this stuff but PDF is easily possible. I put an example in the folder under economics_examples.jmd.

Specifically:

1. There's great LaTeX support for things like numbered systems of equations, referencing equations by label, splitting equations over several lines using align, etc.

2. Writing to HTML is flawless. Writing to Jupyter doesn't chunk out the LaTeX, so you'd need to post-process it. The tangled option works great (just spits out all the Julia code as if it was in one REPL), so we could run that to test.

3. There's support for "silent" code (like boilerplate for setup). There's also support for capturing plot output and resizing it.

4. Headers and footers work great; it's just editing HTML or TeX, using the variables we define in the .jmd and the syntax for calling them given in mustache.jl.

The story on PDF, I think, is twofold. First, there's running it straight with "md2pdf", which I believe fails partly (mainly?) because LaTeX needs to be in shell-escape mode (something to do with minted). I guess we could hack a post-process to enable that (I think you just add a brief note at the top of the document).

If you try and download the PDF from a notebook, it complains because nbconvert can't find Inkscape, which is used to convert SVG images to PDF. People have reported success by installing inscape using Conda, changing the svg2pdf.py file inside nbconvert to point to the install, and downloading.

[jlperla]

Sounds great. We can discuss tomorrow. How is it executing the latex? Can you pass in command line parameters? I suspect there is a way to get the shell escape working...

[arnavs]

Will check re that. I tried running it manually escaped a while back (i.e., go to the repo, do pdflatex [shell-escape flag] ...). I suspect there may be some other roadblocks, but this is the nearest one.

[jlperla]

@Nosferican this is where we discussed some of this.

A few thoughts:

• with something like https://github.com/mpastell/Weave.jl/pull/126 the idea is that convert code blocks to jldoctests and use the typical tools for running those with Documenter.jl, which would give anyone contributing code a proper regression test (perhaps even under Travis for a PR, which would be awesome).
• the preprocessing could also strip or modify equation numbering for jupyter (or convert to something consistent with https://github.com/ipython-contrib/jupyter_contrib_nbextensions or similar extensions
• We can maintain a pandoc dependency, which could help with lots of things like bibtex. Also minted, xelatex, and whatever else makes out lives easier.
• for the PDF output, lets focus on generating the latex itself which is solid (checking the math environments carefully, since the current example doesn't work with the align used right now. There may be some fixes we need to add to Weave.jl to make that stuff work well.
• I can't remember which tex output worked well for me, but try all of the ones listed in http://weavejl.mpastell.com/stable/usage/#Supported-formats-1 called pdf or tex. It might have been md2tex. But check the output, as sometimes it has fragments or incorrect math environments