ReScience challenge: resurrecting a 30-year-old Turbo Pascal program for generating Maucha ionic diagrams

[MikeSilberbauer]

Original article: Silberbauer, M.J. and King, J.M., 1991. Geographical Trends in the Water Chemistry of Wetlands in the South-Western Cape Province, South Africa. Southern African Journal of Aquatic Sciences, 17 (1/2), 82–88. PDF URL: Silberbauer-2020.pdf Metadata URL: http://www.riv.co.za/wv/data/Silberbauer-2020.yaml Code URL: https://archive.softwareheritage.org/browse/origin/https://github.com/MikeSilberbauer/maucha-ionic-diagram.git/ Scientific domain: Aquatic Science Programming language: Turbo Pascal Suggested editor:

[oliviaguest]

The info is there, it just needs to be compiled. However we go back to the same problem as above from two weeks ago when I compile the PDF. Can perhaps @rougier or @MikeSilberbauer compile it and see if they also have this problem? Maybe I'm looking at the wrong PDF, but when I run make this is the output: https://github.com/MikeSilberbauer/Silberbauer-2020/blob/master/article.pdf

Am I missing something? Ideas?

[MikeSilberbauer]

@oliviaguest: I think I forgot to make the change to the title in the metadata.yaml file - I manually changed it in metadata.tex, which gets overwritten, if I understand the process correctly. I have now made the change to the title and removed the blank second reviewer from metadata.yaml. Could you please check if the compile works now?

[oliviaguest]

Hmm... @MikeSilberbauer I get this currently, sadly:

(base) MBpro:Silberbauer-2020 olivia$ make
./yaml-to-latex.py -i metadata.yaml -o metadata.tex
/Users/olivia/Code/Silberbauer-2020/article.py:170: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  document = yaml.load(data)
Traceback (most recent call last):
  File "./yaml-to-latex.py", line 88, in <module>
    content = generate_latex_metadata(filename_in, article)
  File "./yaml-to-latex.py", line 49, in generate_latex_metadata
    "".format(filename=filename, _=article, abstract=abstract))
IndexError: list index out of range
make: *** [metadata.tex] Error 1

I'll investigate.

[oliviaguest]

@rougier so this change breaks everything see comment above.

[rougier]

You have to leave the second reviewer empty indeed. If things are borken, maybe the best is to change metadata.tex manually and then you're done.

[MikeSilberbauer]

@oliviaguest @rougier I have used the file renamed metadata_manual.tex in https://github.com/MikeSilberbauer/Silberbauer-2020 to generate ReScience_C_Silberbauer_2020.pdf in the same repository (using Overleaf).

[oliviaguest]

Ah, great, thanks @MikeSilberbauer. I noticed a typo: (see Appendix, Figure 3. Can you fix that and/or explain how you compiled it, please, so I can compile and submit it. 😊

[MikeSilberbauer]

@oliviaguest Thanks for picking up that missing ")". I have corrected it and recompiled in Overleaf. The new PDF file is named ReScience_C_Silberbauer_2020.pdf in https://github.com/MikeSilberbauer/Silberbauer-2020

[oliviaguest]

Great! @MikeSilberbauer can you either explain how you compiled it and/or fix the date to be today's date for publication? And then I'll try and get it done today, otherwise you might need to put tomorrow's date. 😅

[MikeSilberbauer]

@oliviaguest No stress: I have changed the date to tomorrow in the metadata.tex file in the Overleaf version of the publication, generated the PDF file and copied it as ReScience_C_Silberbauer_2020.pdf in https://github.com/MikeSilberbauer/Silberbauer-2020

[oliviaguest]

@rougier what am I doing wrong now? 😆 😭

(base) MBpro:articles olivia$ ./publish.py --sandbox --metadata ../Silberbauer-2020/metadata.yaml --pdf ../Silberbauer-2020/ReScience_C_Silberbauer_2020.pdf 
Uploading content to Zenodo... Traceback (most recent call last):
  File "./publish.py", line 204, in <module>
    upload_content(server, token, article_id, article_file)
  File "./publish.py", line 25, in upload_content
    raise IOError("%s: " % response.status_code + response.json()["message"])
OSError: 404: PID does not exist.

On Linux too:

(base) olivia@olivia-HP-Z2-Tower-G4-Workstation:~/Code/articles$ ./publish.py --sandbox --metadata ../Silberbauer-2020/metadata.yaml --pdf ../Silberbauer-2020/ReScience_C_Silberbauer_2020.pdf 
Uploading content to Zenodo... Traceback (most recent call last):
  File "./publish.py", line 204, in <module>
    upload_content(server, token, article_id, article_file)
  File "./publish.py", line 25, in upload_content
    raise IOError("%s: " % response.status_code + response.json()["message"])
OSError: 404: PID does not exist.

[rougier]

It depends when you reserved the DOI on Zenodo. (sandbox). You need first to reserve the DOI sung the process, update the PDF and then publish it.

[rougier]

Best is to have the latex sources ready and change the DOI once you get it, recompile and than publish it. Note that you'll also need to update the metadata.yaml (even if you don't use it to re-generate the metadta.tex)

[oliviaguest]

How do you do generate the PDF without using metadata.yaml?

[oliviaguest]

Also I'm confused @rougier... do you mean reserve another DOI? Because I already did that once.

[rougier]

You edit the metadata.yaml with the proper DOI and you also edit the metadata.tex manually (just in cas you have problem going from yaml to tex). For the SANDBOX Zenodo, you can request another DOI and try with this one. These DOI are erased every day I think.

[otizonaizit]

@oliviaguest : if you want we could try to look at the whole process together and see what does and does not work? I have not experienced so many issues as you seem to have while using the new publication process, so there may be some implicit step which is not listed in the docs? Or there may be some implicit software dependencies which are by chance present on my sytem and absent from yours? In any case, publishing should not be such a pain, so I think it's worth it to try to improve the process/the docs.

[oliviaguest]

Hmmm, I am sure @rougier I reserved a real DOI without sandbox.

[oliviaguest]

@otizonaizit sure, over video? when is good for you? I'm on UTC + 3.

[oliviaguest]

@otizonaizit sadly I still have the same problem after requesting a new DOI! See repo: https://github.com/oliviaguest/Silberbauer-2020/

(base) MBpro:articles olivia$ ./process.py --sandbox --metadata ../Silberbauer-2020/metadata.yaml --pdf ../Silberbauer-2020/article.pdf
Article ID:  663987
Article DOI: 10.5072/zenodo.663987
Article URL: https://sandbox.zenodo.org/record/663987/files/article.pdf
(base) MBpro:articles olivia$ ./process.py --zenodo --metadata ../Silberbauer-2020/metadata.yaml --pdf ../Silberbauer-2020/article.pdf
Article ID:  3996198
Article DOI: 10.5281/zenodo.3996198
Article URL: https://zenodo.org/record/3996198/files/article.pdf
(base) MBpro:articles olivia$ ./publish.py --sandbox --metadata ../Silberbauer-2020/metadata.yaml --pdf ../Silberbauer-2020/article.pdf
Uploading content to Zenodo... Traceback (most recent call last):
  File "./publish.py", line 204, in <module>
    upload_content(server, token, article_id, article_file)
  File "./publish.py", line 25, in upload_content
    raise IOError("%s: " % response.status_code + response.json()["message"])
OSError: 404: PID does not exist.

[otizonaizit]

@oliviaguest : what happens if you try the publish.py script directly against the official zenoo and not against the sandbox? The zenodo sandbox has not work reliably for me either in the past. Given that the PDF is now correct and the metadata also, I think there's no risk in trying to do the publication on zenodo directly. If that fails too, maybe we should have another video-chat and see if this is indeed a mac vs Linux issue...

[oliviaguest]

OK, I think that worked!