Fail deployment job when error occurs

[rikhuijzer]

I love the fact that Franklin can generate my website including plots and tables. However, this brings in the risk that some code contains a bug which then shows a broken website. In the worst case, the website would report a stack trace to the reader.

To avoid this, I'm grepping the log and failing the deployment job when the output contains 'Franklin Warning':

    - name: Build site
      run: julia -e '
        using Pkg; Pkg.add(["NodeJS", "Franklin"]);
        using NodeJS; run(`$(npm_cmd()) install highlight.js`);
        using Franklin;
        Pkg.activate("."); Pkg.instantiate();
        optimize();
        verify_links()' > build.log

    - name: Validate output
      run: |
        if grep -1 "Franklin Warning" build.log; then
          echo "Franklin reported a warning"
          exit 1
        else
          echo "Franklin did not report a warning"
        fi

Unfortunately, this doesn't fail on missing images because they only show

 // Image matching '/assets/posts/random-forest/code/u-class.svg' not found. //

in the HTML. I could now also add a grep over the output html pages, but maybe it's better to move this logic into Franklin so that others can also easily use it.

[StefanKarpinski]

Seems like warnings should be fatal, at least in modes that are used to deploy sites.

[rikhuijzer]

@tlienart, in response to your question about my thoughts on what should fail / warn / do nothing in #877:

I think that the main important thing is like what Christopher Rackauckas said about managing large software repositories at Discourse: basically, you want people to be able to review PRs in 5 minutes in the train and merge. So, if CI passes, then the reviewer needs to be able to trust that the website is valid. This is nearly impossible to do in all cases, especially for plots, but I agree with Stefan Karpinski above that deployments should fail whenever something appears to be broken. Specifically, fail on

• missing images,
• errors during code execution,
• user defined assertion fails,
• missing cross-references,
• broken (internal) links, and
• probably more.

These are just my two cents. Note that a bunch of people who interacted with this thread have way more programming experience than me.