search

MarcusBarnes / mik

The Move to Islandora Kit is an extensible PHP command-line tool for converting source content and metadata into packages suitable for importing into Islandora (or other digital repository and preservations systems).
GNU General Public License v3.0
34 stars 11 forks source link

Simple archive toolchain #506

Closed bondjimbond closed 4 years ago

bondjimbond commented 4 years ago

Github issue: #501

What does this Pull Request do?

Creates a toolchain for processing Simple Archive exports from DSpace.

What's new?

New fetcher, writer, filegetter, metadataparser.

Also there's an OAI-PMH fix in there that I couldn't get rid of, but it works.

How should this be tested?

Interested parties

@mjordan @MarcusBarnes

bondjimbond commented 4 years ago

Travis failures are from Travis and not my code, as far as I can tell

bondjimbond commented 4 years ago

Sooooo - any issues with it @mjordan @MarcusBarnes?

mjordan commented 4 years ago

@bondjimbond I'm currently on vacation but I'll smoketest this PR. Sorry for the delay.

mjordan commented 4 years ago

Travis is failing on PSR2 code style: https://travis-ci.org/MarcusBarnes/mik/jobs/586106336.

Can you run phpcs on all new files using vendor/bin/phpcs --standard=PSR2 src/metadataparsers/templated/Xslt.php (using each new filename) and fixing the code style issues?

bondjimbond commented 4 years ago 
FILE: .../brandon/sfuvault/mik/src/fetchers/Filesystem_Subdirectories.php
----------------------------------------------------------------------
FOUND 1 ERROR AFFECTING 1 LINE
----------------------------------------------------------------------
 5 | ERROR | Class name "Filesystem_Subdirectories" is not in camel
   |       | caps format
----------------------------------------------------------------------

If I change this, will I also need to change the filename?

MarcusBarnes commented 4 years ago

@bondjimbond It is best to change the file name to reflect the class name. This is to help keep things consistent with the rest of the codebase and also to adhere to the autoloading PSR-4 convention https://www.php-fig.org/psr/psr-4/ . I note that we don't mention the autoloading standard in the developer page on the wiki.

bondjimbond commented 4 years ago

@MarcusBarnes @mjordan All right, everything passes the sniffs now. If testing, you will have to change the .ini file under Fetcher so that class = FilesystemSubdirectories.

bondjimbond commented 4 years ago

PHP 5.5 is failing because of Travis settings:

5.5 is not pre-installed; installing
Downloading archive: https://storage.googleapis.com/travis-ci-language-archives/php/binaries/ubuntu/16.04/x86_64/php-5.5.tar.bz2
0.12s$ curl -s -o archive.tar.bz2 $archive_url && tar xjf archive.tar.bz2 --directory /
bzip2: (stdin) is not a bzip2 file.
tar: Child returned status 2
tar: Error is not recoverable: exiting now
0.00s0.02s$ phpenv global 5.5
rbenv: version `5.5' not installed
The command "phpenv global 5.5" failed and exited with 1 during .
mjordan commented 4 years ago

Try removing PHP 5.5 from .travis.yml and see if that works.

bondjimbond commented 4 years ago

Travis passing now.

mjordan commented 4 years ago

OK, will test now. @MarcusBarnes are you OK with not testing PHP 5.5?

mjordan commented 4 years ago

Looks good!

Output:

 ajp-v4-id1057.pdf                                                   'MEL Quant.pdf'
 ajp-v4-id1057.xml                                                   'MEL Quant.xml'
'Breastfeeding the late preterm infant.Dosani et al., 2017.pdf'       mik.log
'Breastfeeding the late preterm infant.Dosani et al., 2017.xml'       problem_records.log
 Currie_et_al-2018-BMC_Nursing.pdf                                   'Scaffolding 2014.pdf'
 Currie_et_al-2018-BMC_Nursing.xml                                   'Scaffolding 2014.xml'
'Grounded Theory 2014.pdf'                                            Sonya_Jakubec_Final.pdf
'Grounded Theory 2014.xml'                                            Sonya_Jakubec_Final.xml
'Highly Relevant Mentoring as a Faculty Development Model 2012.pdf'  'Student & Faculty Satisfaction with a Pedagogical Framework 2010.pdf'
'Highly Relevant Mentoring as a Faculty Development Model 2012.xml'  'Student & Faculty Satisfaction with a Pedagogical Framework 2010.xml'
'MEL Qual.pdf'                                                        
'MEL Qual.xml'

mik.log:

[2019-09-26 06:49:51] config.INFO: MIK Configuration {"config_id":"MRU migration"} []
[2019-09-26 06:49:51] config.INFO: MIK Configuration {"last_updated_on":"2019-06-05"} []
[2019-09-26 06:49:51] config.INFO: MIK Configuration {"last_update_by":"bw"} []
[2019-09-26 06:49:58] config.INFO: MIK Configuration {"config_id":"MRU migration"} []
[2019-09-26 06:49:58] config.INFO: MIK Configuration {"last_updated_on":"2019-06-05"} []
[2019-09-26 06:49:58] config.INFO: MIK Configuration {"last_update_by":"bw"} []
[2019-09-26 06:49:58] Info.INFO: MIK started running September 26, 2019, 6:49 am  [] []
[2019-09-26 06:49:58] Info.INFO: Completed in 0.0013456344604492 minutes.  [] []

MODS XML files look good, but I didn't validate them.

Only thing that should be changed is that there is a var_dump() in filegetters/SimpleArchive.php that should be removed. Also, this toolchain should be documented on the wiki, but I don't think we should hold up merging this PR without it, as long as we do it soon.

mjordan commented 4 years ago

@MarcusBarnes if you're OK with merging before documentation is ready, and OK with dropping testing of PHP 5.5, I'll merge after @bondjimbond removes the var_dump().

bondjimbond commented 4 years ago

var_dump removed.

Any thoughts about how to deal with compound objects using this toolchain?

mjordan commented 4 years ago

None other than the ideas culminating in https://github.com/MarcusBarnes/mik/issues/501#issuecomment-528416233. It would be great if only the filegetter and writer needed to be modified to detect and write out compound objects.

bondjimbond commented 4 years ago

@mjordan I'll look at that again once this is merged.

bondjimbond commented 4 years ago

So @mjordan is it merge-ready?

mjordan commented 4 years ago

As soon as I hear from @MarcusBarnes on dropping testing in travis.yml for PHP 5.5.

bondjimbond commented 4 years ago

@MarcusBarnes Any complaints about dropping PHP 5.5 from Travis?

bondjimbond commented 4 years ago

@MarcusBarnes @mjordan This has been sitting around a while.

mjordan commented 4 years ago

I say let's merge it. Do docs later.

bondjimbond commented 4 years ago

I'm happy with that decision! I think the only question here was whether it was OK to drop PHP 5.5 testing from Travis; we never heard from @MarcusBarnes on that.

MarcusBarnes commented 4 years ago

@bondjimbond I'm OK with dropping PHP 5.5 from our Travis checks as it's long past end-of-life with regards to PHP support. Those using MIK can use it from older commits. At some point, we should cut another release to capture the state of MIK before adding additional changes. Thanks again @bondjimbond @mjordan for all your work on MIK!

bondjimbond commented 4 years ago

Thanks, Marcus. @mjordan I think this means you're clear to merge. :)