Complete the offline renderer

[joeyespo]

The --render-offline didn't make the 2.0 release, because it doesn't render the same content as GitHub.

The functionality is there. The CLI option is added back again in the fix-render-offline branch, where work can continue.

[joeyespo]

Relabeling this as an enhancement because this was never completed in the first place.

[GochoMugo]

it would be helpful if the Markdown documents could be rendered offline

[joeyespo]

Just a small update / thinking out loud.

Now that Grip has tests (#150) which includes a Markdown file containing all the features of GitHub Flavored Markdown and its GitHub API output, ensuring the correctness of an offline renderer implementation should be easier.

Once it's complete, it can become the default (although the styles and assets should still come from GitHub). The GitHub API can still be used to access Markdown features that Grip's offline renderer doesn't implement right away.

[mileserickson]

I just received a "github API rate limit reached" error randomly: it's my first time running grip for the day and I wanted to view a single document, but I'm at a busy tech campus where 100+ people are using Github on a routine basis.

I hadn't even realized that Grip didn't render offline. I'm confused as to why I would use Grip at all, if it's no more efficient than viewing the Markdown document in the original repo. Seems like a high priority.

[joeyespo]

@mileserickson I understand your frustration.

The reason it works this way is that from the beginning, correctness of rendering was chosen over offline availability. It's absolutely a limitation, but with the recent test improvements in #148 and 2c37cc455ad34a0b4a05482d31f7f7877ee58c59, it'll be easier to make sure correctness is preserved in an offline renderer.

I'm confused as to why I would use Grip at all, if it's no more efficient than viewing the Markdown document in the original repo.

Grip solves the problem of viewing Markdown before committing to the repo. Offline mode solves the additional problem of viewing without an internet connection. (That's still necessary for viewing the Markdown document in the original repo, btw).

I'm planning on addressing this at some point, unless someone else jumps in. However, I have fewer large windows of time for big changes these days. Being aware of that actually influenced the original decision. (Nobody likes waiting for a busy maintainer to change and deploy code every time GitHub tweaks or updates their styles.)

In the meantime, could you try the workaround? That'll prevent others on your network from using up your own rate limit. Here's how:

1. Generate a Personal Access Token

   

2. Copy the token

   

3. Create a ~/.grip/settings.py file:

   $ cd ~
   $ mkdir .grip
   $ cd .grip
   $ touch settings.py

4. Add the token to your new settings.py file:

   PASSWORD = 'YOUR-ACCESS-TOKEN'

Eventually this auth experience will be also improved (see #82).

Thanks for trying out Grip and for being patient. Let me know if I can help in any other way.

[z]

The solution above worked for me.

[astrodad]

Is it still the plan to implement offline renders? I don't need a perfect render, just something usable for reading markdown books.

[joeyespo]

@astrodad It is. For me personally, I probably won't start until late summer or in the fall.

Thanks for checking in!

[astrodad]

Great. I'll take a look at the code but my Python is as good as my mastery of the Force.

[Peque]

Any updates on this matter? :blush:

[Oneoneonder]

What is holding up the offline renderer?

[hollasch]

I just hit the rate limit, and followed the instructions above to set up a personal access token, which worked great. True offline access would be great, but this is a fine workaround. Am I correct in thinking that this would be the superior solution to supplying --user and --pass on the command line? If so, it seems that grip/templates/limit.html should be updated.

If you'd like, I can create a separate issue. Also, I can take this on and issue a PR when ready.

[rsyring]

FWIW, I wonder if the initial offline renderer could be documented as not-exact. That is, for the sake of getting offline rendering off the ground, just state that the offline renderer won't be perfect. My guess is that many of the people needing that support don't care that it isn't exact, they will be happy to just have offline rendering that is close.

Then, over time, the offline renderer could be brought up to par.

[facelessuser]

It's going to be hard to find a render that does things exactly offline like GitHub unless you want to use their Markdown parser...which I think is Redcarpet? Maybe I'm wrong.

You can get githubish though with Python Markdown. I have a branch over here where I enabled offline mode: https://github.com/facelessuser/grip. It's using my dev branch of extensions here: https://github.com/facelessuser/pymdown-extensions/tree/dev. These extensions don't aim for 1:1 exactness for Github (nor do I think I would ever go that far due to inherent differences in parsers), but much of it is inspired by Github. It handles emojis (uses a literal index from their Gemoji repo), githubish emphasis, tasklists, auto-linking links and emails etc. It's more than workable for a test offline rendering. Just saying there are closish things.

Not saying you should use anything from here, just letting you know some options that are available.

Anyways, this is a nice project that I've started using on my Github readmes and such.

EDIT: If I had a place to upload the results I would.

[feinstaub]

First of all, thanks for grip. I just discovered it and now I use this script to use the offline renderer:

#!/usr/bin/python3
from grip import serve
serve(render_offline=True)

I use python 3.6. To make this script work I have to comment out 4 lines from line 122 in ~/.local/lib/python3.6/site-packages/grip/renderers.py

        #if markdown is None:
            #import markdown
        #if UrlizeExtension is None:
            #from .mdx_urlize import UrlizeExtension

Is this normal?

The most obvious rendering problem is that with lists there must be a preceding empty line. This also results in TOCs without indentation. Any idea for a quick fix?

[facelessuser]

It is a limitation (or I should say according to spec) of the Python Markdown library which is used to render the HTML from Markdown. There is no easy way fix/change. You would have to override the default extensions to rework then in a way that is closer to GFM.

To be honest closer is all I think you could ever get with Python Markdown. You can achieve GitHub-ish results, but you will never get 1:1 results. For 1:1 results in Python, you would have to port GFM directly to Python.

[feinstaub]

@facelessuser thanks for the assessment. Then I think it will be better to change https://github.com/jonschlinkert/markdown-toc to produce a Python Markdown compatible TOC output.

[yarwelp]

@facelessuser

To be honest closer is all I think you could ever get with Python Markdown. You can achieve GitHub-ish results, but you will never get 1:1 results. For 1:1 results in Python, you would have to port GFM directly to Python.

I agree. I did a little bit of searching and found a question on SO about GFM in Python with an accepted answer that states the following:

GitHub uses Redcarpet which is really a Ruby binding built by github for Sundown.

There's a binding for Sundown in Python called Misaka. If you want to have something closer to the renderer github uses but in python you might start there.

For code highlighting you can use Pygments with Misaka.

So in case anyone has the time, perhaps it would be fruitful for them to look into Misaka? The official repo for Misaka is on GitHub; https://github.com/FSX/misaka.

The GitHub repo description of Misaka reads "A Python binding for Hoedown". Hoedown, in turn (also on GitHub) is described in the README as "a revived fork of Sundown".

Misaka is published on PyPi, so it can be installed simply by issuing pip install misaka, provided that one has Hoedown installed.

[yarwelp]

If anyone decides to go the Misaka route, note also one of the comments on the accepted answer in the SO thread:

This is how I made it work. If someone else is planning to use misaka, please beware of this

The issue that is pointed out was closed but doesn't link a commit. I would assume that it means that the XSS issue has been resolved but rather than assume such to be the case one should test and see that it really is.

[ewetmill]

I use python 3.6. To make this script work I have to comment out 4 lines from line 122 in ~/.local/lib/python3.6/site-packages/grip/renderers.py

    #if markdown is None:
        #import markdown
    #if UrlizeExtension is None:
        #from .mdx_urlize import UrlizeExtension

Is this normal?

I have the same problem with python 2.7

[davestewart]

It's the end of 2018, is there any progress on this?

[OleMussmann]

@davestewart The API rate limit bit me repeatedly - especially when trying to 'live edit' a markdown file - so I wrote a markdown converter that also works offline. It's not pixel-perfect compared to grip, but it comes close.

It's called marge and you can install it via snap, ppa, deb, or tarball. Code is on gitlab. Be aware that it's a rather fresh project, there might still be occasional hiccups.

@joeyespo Feel free to snatch code from me for grip's offline-mode. If you can settle for less than 100% conformity with online-grip then I can help you with the implementation. Please also let me know if you would like me to remove this shameless plug.

[davestewart]

@OleMussmann - I created a ticket on GitLab.

[ianfixes]

Subscribing for interest in this feature.

As far as an MVP of offline rendering goes, my usage of this program is to export README.md to README.html as a precursor to using README.html as an "overview" for the generated documentation for a project. (i.e. groovydoc accepts an HTML document to be merged into the index.html for generated documentation). Pixel-perfect rendering and syntax highlighting isn't required, just a basic conversion of markdown into HTML.

[hollasch]

@ianfixes — for HTML documentation, I strongly encourage you to check out Markdeep. In addition to standard Markup, it supports diagrams, mathematical notation, and a whole host of very useful additions.

For an example, the Ray Tracing in One Weekend series is written entirely in Markdeep: https://raytracing.github.io/. Not only do you have a much larger palette of documentation features, you also have complete freedom to style as necessary with standard CSS.

Also, take a look at the style/book.css — it includes media-print rules for PDF output.

[ianfixes]

That seems interesting, but anything beyond what Github would be rendering on their site is wasted in my particular use case.

I ended up using GitHub's own ruby gem called redcarpet:

require 'redcarpet'

puts "Reading README.md"
input = File.read("README.md")

puts "Initializing GFM -> HTML converter"
renderer = Redcarpet::Render::HTML.new(
  no_images: true,
#  no_styles: true,
)
markdown = Redcarpet::Markdown.new(
  Redcarpet::Render::HTML,
  no_intra_emphasis: true,
  autolink: true,
  tables: true,
  fenced_code_blocks: true,
  lax_spacing: true,
)

puts "Rendering markdown to HTML in #{OVERVIEW_FILE}"
intro = "via [README.md](https://github.com/my/repo):\n"
output = markdown.render(intro + input)
File.write(OVERVIEW_FILE, output)

[Boscop]

Any update on this? :) I'm often getting rate-limited, even when using --user and --pass, even though that was my first try and I wasn't hitting any limits myself. Weird..

[Asday]

Is there any reason the API-returned data isn't simply cached? I don't see why it needs to be loaded for every single page render.

[AlexDev404]

Is it still the plan to implement offline renders? I don't need a perfect render, just something usable for reading markdown books.

@astrodad Try using something like Typora, it gives you a preview of how it will look like. I just use Grip in conjunction with Typora because they really work well together with Typora being the editor and Grip being the viewer/exporter.

[davestewart]

I can confirm that Typora is a bloody brilliant MD editor:

• https://typora.io/

Their GitHub theme needs a bit of work (it's not exact) but is pretty close.

You can also set up your own theme (I couldn't be bothered TBH)

[MarkLodato]

In case anyone is looking to implement this, GitHub's renderer is available at https://github.com/github/cmark-gfm and there are various Python bindings. You'll have to figure out the right combination of flags, something like cmark-gfm -e table -e autolink --github-pre-lang. You'll also need to get the CSS correct. But perhaps this is enough to get someone started.

[wpyoga]

Is there any reason the API-returned data isn't simply cached? I don't see why it needs to be loaded for every single page render.

I personally think this would be easy-ish to implement and won't break too many things, will it? The cache can be refreshed when it gets older than an hour or a day maybe. And it can also be forcefully refreshed if the user wishes to do so.

[Asday]

Personally I'd argue for never updating the cache unless the user requests it, but yeah your thoughts match mine.

[ruslanzelinskyy]

Ended up with uninstalling grip and using glow

[justincbagley]

I have been using grip for markdown preview, conversion, and PDF saving on macOS for years. So thanks.

However, lately, I've started running into GitHub API limits. I found this quite interrupting of my flow, so I looked into other options. I learned that Atom has a nice Markdown preview package built in now (available by default, and on GitHub here).

So, my current approach is to build my GitHub-flavored Markdown document in Atom with the preview function running, then to do a 'final' run with grip and save as PDF from my browser window (e.g. Safari).

I also tried glow (mentioned above). It is cool but not as nice looking and doesn't really seem to work well for my purposes, so I can't recommend it.

Additionally, I tried the editR R package with poor results. I could get the shiny app to run but I could not get it to correctly display my markdown, so I cannot recommend it.

Hope this helps someone. ~J

[MapleCCC]

Thanks to @justincbagley for elaborating on his experience in trying several other options and kindly sharing with the community.

Unfortunately, it looks like grip is abandoned. The last commit is three years ago. And the author's GitHub account is inactive for quite some time. Sadly this happens to more than one open source projects I have been using and liking. Maybe it's time to move on. If someone is interested in taking the maintenance job and making a fork, all discussions can be transferred there. Transferring the project to jazzband is yet another option.

[Zeioth]

It's a shame, this is a really neat package when combined with vim-grip

[fpf3]

I just use pandoc now. Get the style sheet for github, and use it with vim/zathura. Bind f5 to rebuild the document -------- Original message --------From: Zeioth @.> Date: 12/17/21 10:16 (GMT-05:00) To: joeyespo/grip @.> Cc: Fred Frey @.>, Manual @.> Subject: Re: [joeyespo/grip] Complete the offline renderer (#35) It's a shame, this is a really neat package when combined with vim-grip

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Zeioth]

In the end I took this strategy on vim-grip:

nnoremap <C-n> :GripStart<CR>:sleep 1<CR>:GripStop<CR>

This runs grip to render the document in the browser, and then stops grip. This way even 60 requests is enough. My main problem was that grip is real-time and I was running into the rate limit after a couple minutes.

[bitti]

I ended up using GitHub's own ruby gem called redcarpet

That's actually an underrated comment. Accompanied with the right CSS this comes pretty close to the original. Only if there was a way to extract all the necessary CSS from GitHub. Luckily there is already a project doing exactly that: https://github.com/sindresorhus/github-markdown-css.

So inspired by this I hacked together a script which pretty much replaces grip for me: https://gist.github.com/bitti/016ea088f1e9dfc7120eb5f8c937c2da. Even though it's a hack I think it works pretty well, as long as you don't also need syntax highlighting (which I suppose would be much more work, maybe by integrating PrettyLights or starry-nights?).

[jchook]

The --render-offline didn't make the 2.0 release, because it doesn't render the same content as GitHub.

If it's close enough for 95% of cases maybe ship it with a warning?

[Bandisa]

This issue has been open for 10 years now, will this ever be resolved?

[buhtz]

• How to Contribute to Open Source
• Open source runs on non-code contributions
• Revitalizing stalled open source projects
• Avoiding common pitfalls when first contributing to open source - Tips and tricks for getting started
• 5 Ways to Get Started in Open Source
• How to contribute to open source
• How to contribute to Codeberg.org

[Asday]

This issue has been open for 10 years now, will this ever be resolved?

Put less flippantly - yes, as soon as someone works on it. Given that it's been ten years as you rightly noted, one can assume that there's not much interest in the feature and/or project.

[Bandisa]

Cool, makes sense. I'm interested, so I'll attempt to implement this within the next 2 months. Thank you.

Edit: Apologies for my tone also, my comments thus far haven't been nice nor competent.

[Bandisa]

Update

I have investigated the feasibility, not sure if it will be the best approach but I think for an mvp I can make it work.

I built a small python script that runs node as a subprocess in a separate thread. The nodejs script uses https://github.com/markdown-it/markdown-it to render the markdown file and outputs a output.html file which seems to render the file offline successfully.

I use vscode-github-markdown-preview-style extension to render .md files on vs-code and upon going through the code base, I discovered that they have a very good css github theme that I built locally and added to the render.

Now this is more like hot wiring and would probably be best to just use a nodejs server directly but as a proof of concept it seems enough.

Will dive deeper, design and implement this properly soon so that I can open a PR and get insights from those who are interested.

Results

output.html