search

openlibhums / imports

A plugin for importing content into Janeway.
GNU Affero General Public License v3.0
1 stars 2 forks source link

Article file support #9

Closed mauromsl closed 3 years ago

mauromsl commented 3 years ago

Allows importing article galley files with the csv metadata importer.

We can later add support for other schemes (http, ftp etc.)

hachacha commented 3 years ago

Was denied to contribute to this just made a small change in utils.py the N (for corporate) on line 34 for CSV_MAURO needs a comma after it.

I can't get this to upload properly for some reason. Perhaps I'm missing some sort of library?

  1. When I try to upload just the pdf I get image with a import complete green message in bottom right

  2. When I try to upload everything as the example.csv with the two images separated by a pipe in the zip column I get image but they are right here: image

  3. When putting in a zip file image

here is my csv (in the final round, the other times i changed the zip column)

Article identifier, Article title,Section Name, Volume number, Issue number, Subtitle, Abstract, publication stage, date/time accepted, date/time publishded , DOI, Author Salutation, Author first name,Author Middle Name, Author last name, Author Institution, Author Email, Is Corporate (Y/N), PDF URI,XML URI, HTML URI, Figures URI (zip)
80,jk new article,Articles,1,2,the best subtitle you could ever odo have wooo,abstract me right now,Published,2020-01-01T09:00:00,2020-01-02T09:00:00,10.1000/xyz123,Mr,Adi,Amit,Amit,Open University of Israel,adiam@openu.ac.il,N,file:///home/jon/Desktop/ncmr12169.pdf, file:///home/jon/Desktop/ncmr12169.xml,file:///home/jon/Desktop/ncmr12169.html,file:///home/jon/Desktop/17504708_2020_13_4.zip

One feature request and one question: Can we add the author biography field in this? Is there a proper way to escape commas that may appear in the abstract? Should they bt put in as html entities or something else? Forward slash doesn't seem to work from my test.

mauromsl commented 3 years ago

Thanks for the feedback @hachacha , I added the field for bio, as well as the missing comma.

In order to include commans in the csv values just enclose the value within quotes ("value, with,commas")

There was a mistake on the example csv headers. For the figures, I settled on using a zipfile instead of a pipe separated list of files. We already have a function for loading up article figures in bulk from a zipfile in Janeway core, which is what this plugin is now using.

So instead, in the figures column, add an URI pointing to a zip file that contains all the images. The name of the figure files must match the src on the html or xlink:href on the JATS XML (as per Janeway's logic)

hachacha commented 3 years ago

Awesome! I ran this on my local and noticed 3 things:

  1. it hung for a while and I looked at the console to see that Pdb was running so I hit continue: 
    INFO 2020-12-02 13:32:32,641 utils P:8382 T:139644866889472 [ETHOS] Writing CSV import errors to /home/jon/repos/janeway/src/files/temp/janeway-imports-63d1d2c5-b03d-451d-a57c-37275c59bd2e.csv
/home/jon/.virtualenvs/janeway/lib/python3.6/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Article.date_accepted received a naive datetime (2020-01-01 09:00:00) while time zone support is active.
RuntimeWarning)
/home/jon/.virtualenvs/janeway/lib/python3.6/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Article.date_published received a naive datetime (2020-01-02 09:00:00) while time zone support is active.
RuntimeWarning)
> /home/jon/repos/janeway/src/plugins/imports/utils.py(212)import_author()
-> salutation, first_name, middle_name, last_name, institution, bio, email = author_fields
(Pdb) continue

and it finished. properly. Is this only because I'm running in dev mode?

  1. When looking at the finished article, the doi is not being placed in the publication info and it is instead defaulting to the next available ID. image

  2. I get a joined path error for the xsl file when trying to view the article. Assumed this was because there were not files that were set for production but that issue still persisted after trying to add it and there is some other strangeness with editing files I have a feeling this might have something to do with my local set up.

Request Method: | GET
-- | --
http://localhost:8000/ETHOS/article/id/60/
1.11.29
SuspiciousFileOperation
The joined path (/home/jon/repos/janeway/src/transform/xsl/default_w4gNICq.xsl) is located outside of the base path component (/home/jon/repos/janeway/src/files/xsl)
/home/jon/.virtualenvs/janeway/lib/python3.6/site-packages/django/utils/_os.py in safe_join, line 78
/home/jon/.virtualenvs/janeway/bin/python
3.6.7
['/home/jon/repos/janeway/src',  '/home/jon/.virtualenvs/janeway/lib/python36.zip',  '/home/jon/.virtualenvs/janeway/lib/python3.6',  '/home/jon/.virtualenvs/janeway/lib/python3.6/lib-dynload',  '/usr/lib/python3.6',  '/home/jon/.virtualenvs/janeway/lib/python3.6/site-packages',  '/home/jon/.virtualenvs/janeway/src/django-bootstrap4',  '/home/jon/.virtualenvs/janeway/src/foundationform',  '/home/jon/.virtualenvs/janeway/src/django-mailgun',  '/home/jon/.virtualenvs/janeway/src/django-simple-math-captcha',  '/home/jon/.virtualenvs/janeway/src/dynamicsites',  '/home/jon/.virtualenvs/janeway/src/ebooklib',  '/home/jon/.virtualenvs/janeway/src/django-materialize',  '/home/jon/repos/janeway/src/plugins']
Wed, 2 Dec 2020 13:59:16 -0600

image

mauromsl commented 3 years ago

Awesome! I ran this on my local and noticed 3 things:

1. it hung for a while and I looked at the console to see that Pdb was running so I hit continue:
INFO 2020-12-02 13:32:32,641 utils P:8382 T:139644866889472 [ETHOS] Writing CSV import errors to /home/jon/repos/janeway/src/files/temp/janeway-imports-63d1d2c5-b03d-451d-a57c-37275c59bd2e.csv
/home/jon/.virtualenvs/janeway/lib/python3.6/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Article.date_accepted received a naive datetime (2020-01-01 09:00:00) while time zone support is active.
  RuntimeWarning)
/home/jon/.virtualenvs/janeway/lib/python3.6/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Article.date_published received a naive datetime (2020-01-02 09:00:00) while time zone support is active.
  RuntimeWarning)
> /home/jon/repos/janeway/src/plugins/imports/utils.py(212)import_author()
-> salutation, first_name, middle_name, last_name, institution, bio, email = author_fields
(Pdb) continue

and it finished. properly. Is this only because I'm running in dev mode?

1. When looking at the finished article, the doi is not being placed in the publication info and it is instead defaulting to the next available ID.
   ![image](https://user-images.githubusercontent.com/1020384/100923399-bb80e680-34a4-11eb-8518-cb4d5b2f6487.png)

2. I get a joined path error for the xsl file when trying to view the article. Assumed this was because there were not files that were set for production but that issue still persisted after trying to add it and there is some other strangeness with editing files
   I have a feeling this might have something to do with my local set up.
Request Method: | GET
-- | --
http://localhost:8000/ETHOS/article/id/60/
1.11.29
SuspiciousFileOperation
The joined path (/home/jon/repos/janeway/src/transform/xsl/default_w4gNICq.xsl) is located outside of the base path component (/home/jon/repos/janeway/src/files/xsl)
/home/jon/.virtualenvs/janeway/lib/python3.6/site-packages/django/utils/_os.py in safe_join, line 78
/home/jon/.virtualenvs/janeway/bin/python
3.6.7
['/home/jon/repos/janeway/src',  '/home/jon/.virtualenvs/janeway/lib/python36.zip',  '/home/jon/.virtualenvs/janeway/lib/python3.6',  '/home/jon/.virtualenvs/janeway/lib/python3.6/lib-dynload',  '/usr/lib/python3.6',  '/home/jon/.virtualenvs/janeway/lib/python3.6/site-packages',  '/home/jon/.virtualenvs/janeway/src/django-bootstrap4',  '/home/jon/.virtualenvs/janeway/src/foundationform',  '/home/jon/.virtualenvs/janeway/src/django-mailgun',  '/home/jon/.virtualenvs/janeway/src/django-simple-math-captcha',  '/home/jon/.virtualenvs/janeway/src/dynamicsites',  '/home/jon/.virtualenvs/janeway/src/ebooklib',  '/home/jon/.virtualenvs/janeway/src/django-materialize',  '/home/jon/repos/janeway/src/plugins']
Wed, 2 Dec 2020 13:59:16 -0600

image

Thanks @hachacha

Added a few more improvements:

Regarding the XSLT error, its not related to this pr but it looks like for some odd reason Janeway its trying to find the stylesheet in the wrong place. They should be under src/files/xsl.

Can you check that the path is correct under admin/core/xslfile? they should be relative paths, here is how they look like on my dev setup: image