search

NCEAS / metacatui

MetacatUI: A client-side web interface for DataONE data repositories
https://nceas.github.io/metacatui
Apache License 2.0
42 stars 27 forks source link

Implement Markdown View #820

Closed robyngit closed 5 years ago

robyngit commented 5 years ago

The Markdown View will parse markdown contained within the XML associated with each project. This View must be flexible enough to be used as a component in other (non-project) parts of MetacatUI.

This is a subtask of NCEAS/project-papers#35

robyngit commented 5 years ago

This issue is closely related to issues #589 and issue #726. I will record updates in this issue only from now on.

The list of markdown elements that we'd like to support are listed in this Javascript-Markdown-Library-Comparison spreadsheet.

To test a markdown to html converter, I created an example XML doc that includes all of these elements.

I used this xml doc to test the functionality of showdownjs. The resulting demonstration can be viewed in this JSfiddle.

Since showdownjs has almost all of the functionality we require, I will move forward with implementing this in the markdown view.

csjx commented 5 years ago

Awesome @robyngit - your fiddle looks like Showdown is really promising. 🤠

mbjones commented 5 years ago

Fantastic, @robyngit. I really like how thorough your test and demo are. So I see that two major features are not in this demo: math and citations. I'm not clear as to why the math support is hard to test here. Can you explain? And how confident are you that showdown will be able to support those, or that we will be able to write a plugin that supports them? Note that the remark.js supports math through a plugin. Thanks for the clarifications.

csjx commented 5 years ago

@robyngit wrote:

Math should be possible using the showdown-katex extension. However, it's too complicated to demonstrate using this JSfiddle environment (no CDN available).

While JSFiddle doesn't provide a CDN-linked copy of the math or citation showdown JS libraries, you can install the library somewhere web-accessible and link directly to it in JSFiddle. Is that the only issue?

csjx commented 5 years ago

I also noticed that the DocBook tags don't seem to be supported (at least in the fiddle). I would expect the <para> tag to be translated into a <p> tag, but it looks like it is just passed straight through, which is expected (and then sanitized out by the HTML renderer).

So, we may need to do some pre-processing from DocBook to inline HTML so the markdown processor can then do the rest. I'm not seeing much for DocBook to HTML Javascript libraries out there (server side there is pandoc of course). Since EML only supports a small subset of DocBook tags, writing an XSLT stylesheet that does the conversion would be tractable, and could be applied in the browser as a pre-processing step.

robyngit commented 5 years ago

@csjx @mbjones: As Chris suggested, the main problem was a lack of CDN's. I was also getting a Cross-Origin Read Blocking error when trying to link directly to js files hosted elsewhere. (A chrome thing?) From reviewing the documentation for the math and citation extensions, and checking out issues on the respective github repos, I feel pretty confident that showdown will be able to support those features.

You're right, the <para> tags aren't converted, but they do remain in the html. I should have been more clear on that. I think it would also be possible to write a showdown extension that does the conversion to html. For example, see how the 'bindings' and 'codehighlight' extensions are created in the fiddle.

csjx commented 5 years ago

Ah, yes, a showdown extension for Docbook XML would be great.

For the non-CDN hosting, you would need to host the files on a web server that supports cross origin requests (CORS). dev.nceas.ucsb.edu, test.arcticdata.io, etc are set up that way. But like you said, showdown support looks probable, so I don't feel like the fiddle needs to be updated. Doing this work directly in MetacatUI should be fine.

robyngit commented 5 years ago

Great! Then I'll continue moving forward with getting showdown + extensions working in metacatUI. :)

robyngit commented 5 years ago

For reference: The example project XML that includes all the complex markdown is available on dev. knb. Its id is urn:uuid:5a26fb11-e0a1-4e3f-976e-1090f24089c0.

robyngit commented 5 years ago

Showdown is now working in MarkdownView! 🎉 Just need to add in the extensions. This can be previewed with all the markdown elements we will support here:http://<YOUR-LOCAL-METACAT-UI-ADDRESS>/#project/urn:uuid:5a26fb11-e0a1-4e3f-976e-1090f24089c0.

Extensions to add:

robyngit commented 5 years ago

A note: The citation extension retrieves some data from doi.org. Because of a metacat bug in how metadata is submitted when we register a doi, author information for any of our cited datasets may be incorrect or look pretty strange (e.g. author shows up as DC=Dataone CN=SASAP for doi 10.5063/f1r78cf3). Once this issue is resolved, dataset citations should look correct.

robyngit commented 5 years ago

citations in markdown

The last issue to resolve in the markdown view is getting inline citations and bibliographies to work as we would like.

First attempt

Initially, I made an extension that searched for and extracted any DOI, WikidataID, or other input, contained within square brackets like so:

^[doi:xxxxxxxx]

The extension then took this ID and used citation.js to lookup the associated citation information. It then created an appropriate in-line citation in place of the ^[doi:...], and appended a bibliography to the end of the markdown view.

Issues with this solution

  1. There was a noticeable delay in rendering the view, even with only three citations to retrieve. Showdown does not currently support async filters, although this is a pending enhancement
  2. More importantly, we prefer the bibliographic information to be stored in the project xml for long-term archiving purposes.

New strategy

Bibtex style citation information will be stored in the project xml (and thus in the project model). This information will be passed to the citation showdown extension. The extension should search for bibtex keys in square brackets, like so...

[@darwin_1859]

...and match them to the citation information contained within the model. It should use this information to create an in-line citation. The literature cited/bibliography will be created in a separate view.

Progress

laurenwalker commented 5 years ago

I'm gonna close this issue since we have the remaining task detailed in #870.