Automatically update publication list

[alexfoias]

• [ ] Update bibeasy so it can output nice Markdown format. See https://github.com/jcohenadad/bibeasy/pull/18
  • Syntax: bibeasy -o publis.md --type article conf-article --reverse --freshen-cache --combine
• [ ] Add feature to merge multiple gsheets (from Julien, Nikola, Eva, Benjamin)
• [ ] Cleanup gsheets from all PIs
• [ ] Add Github action that runs bibeasy on RTD build.

[jcohenadad]

@alexfoias i think we already have a related issue on another repository— can you please cross reference

[alexfoias]

Yes, I remembered the same thing. A similar issue is https://github.com/spinalcordtoolbox/spinalcordtoolbox/issues/3169

[kousu]

I've been tracking this in https://github.com/jcohenadad/bibeasy/issues/16.

I think solving this with bibeasy will be, well, easy.

[jcohenadad]

@kousu there is already a list of publications for me and Nikola. Benjamin and Eva can create their own ones (or maybe they already have) so the idea (I think) would be to run a script that runs bibeasy on our four google sheets, remove duplicates, and update the publication page. I suppose we can have it run during compilation on RTD?

[kousu]

I would very much like to have it run during site compilation. bibeasy needs to be taught either how to output markdown directly, or output .bib (that was my idea in https://github.com/jcohenadad/bibeasy/issues/16) which we then feed into a .bib -> markdown plugin, or a sphinx .bib plugin. My guess is that .bib will be more future-proof.

[jcohenadad]

Right, I forgot about the .bib solution, which does seem like an elegant solution indeed. I'm just afraid we will run into limitations by not being able to customize the format of the output citations. For example, we might want to use bold characters for journal names, add hyperlink to the article, underline student's name, add custom fields like 'best poster award' or things like that. Also, I'm anticipating that complying to .bib will require some headaches because some fields are not properly describing the structure of a bib citation (eg: column 'volume:page'), whereas to port it into a md page, we don't need to parse this. What do you think?

[kousu]

I think.... we should implement something and just use it and get it working, because it's not something we ever intend to build anything on. So probably straight to markdown.

[namgo]

I've been working on automating the bibeasy->neuro.polymtl.ca system, initially locally with podman. Now that it's working locally I'll be moving things into github actions.

Only modification that I'll need to make in the website's repo is to add the generated filenames to the tocfile list.

I will create a public repo now and move only the latest snapshot of bibeasy to it (avoiding leaky keys).

Julien made an excellent point that the bibeasy import gh-action should be running at a set interval (weekly) rather than on every commit. I don't know how to do this offhand, but it seems very doable so I'll look into it.

[namgo]

https://github.com/namgo/neuro.polymtl.ca I think the gh-action to generate pages with bibeasy is at a decent spot now. Only thing left is to switch it to a weekly job I believe.

[namgo]

Update on the weekly job: unless we cache older gen-{}.md pages in the main repo, we might be best off regenerating the markdown on push:

• There's not that many pushes per month
• The build process is relatively quick
• I don't have a better solution

I've also set up a weekly job, in case there are updates to publications and nobody commits to the website during that week.

[jcohenadad]

This is awesome, @namgo . Where can we see your development? Do you have a working branch or something?

[namgo]

Absolutely, so under https://github.com/namgo/neuro.polymtl.ca I have a added a weekly.yml and modified the publish.yml. https://github.com/namgo/neuro.polymtl.ca/tree/master/.github/workflows

Prior to merging it back, I'll need to re-add the cname definition in both files.

I guess I informally requested a review but didn't specify what I'd like reviewed, whoops! The changes shouldn't break anything now, but I just want to confirm that while the website is important, needing to temporarily revert it if my changes do in fact cause a problem, is not itself a huge cause for concern?

[namgo]

http://test-neuro.nzm.ca is available as an alias to my fork's github.io if anyone wants to check it out in the next few days.

I just want to point out the duplicate/redundant titles under data, these are generated and I don't know what to name them. Should we get rid of any of the existing files?

[jcohenadad]

This is great @namgo !

Just to clarify: was "Journal articles" automatically generated?

Also, you can get rid of "Articles", "Conference Articles" and "Combined" as these are already included in "Journal articles".

[namgo]

Journal articles, Book Chapters, Theses, Public News, Data were there already.

Articles, Conference Articles, Combined, are generated.

I haven't integrated the javascript yet.

[jcohenadad]

i see-- so the "combined" would be the one to use, right?

I haven't integrated the javascript yet.

what javascript needs to be integrated? this?

<div id="filter-box">
</div>

[namgo]

/_static/js/custom.js would put its contents into that div, I believe. That's the javascript I was referring to, yeah.

Oh! I guess gen-combined.md is the only generated file that's needed here. I will: 1) place the filter system into gen-combined.md 2) rename gen-combined.md to articles.md 3) remove gen-articles.md gen-conf-articles.md 4) remove journal_articles.md

https://namgo.github.io/neuro.polymtl.ca/ is the current link (cname setting got overwritten).

[namgo]

I completely misread the original journal_articles.md and the fact that we were including the generated dataset already within that. I've modified journal_articles to include the gen-combined.md - no sense removing it, my mistake.

https://namgo.github.io/neuro.polymtl.ca/publications/journal_articles.html

It basically works now, I'll just remove the modification I made to bibeasy which document title names to generated files.

[jcohenadad]

Fixed in #90