Resources enhancements

[veggiematts]

Structural Changes:

• A resource can have more than one parent

• IsbnOrIssn is repeatable

  New features:

  Import tool:

  This tool allows to import resources and organizations from a CSV file.

• CSV file format: The CSV separator can be comma, semi-column or pipe. The first line must be column headers, not data. Example: titleText,resourceURL,ISSN,EISSN,organization,role,parentResource The user can then choose which columns to use for import. If a column header name matches a database column name, it will be automatically selected accordingly. A CSV file example is attached to this merge request.
• Resources import: Each line is a resource. If there is a parentResource column, for each line processed, the parentResource will be created if it does not exist already, and the processed ressource will be attached to it.
• Organization import: If there is an organization and/or role column, organizations can be imported. This works wether the Organizations module is in use or not. If the role column is not present, a default role will be assigned to the organization.

Step 1: Select the file and the CSV separator.

Step 2: Select which columns are to be imported.

Step 3: Import results.

Repeatable ISBN:

It is now possible to have multiple Isbn or Issn for a resource:

• A resource can have more than one parent:
• Children deletion: The user can delete a parent resource and all its children that don't have another parent at once.

  Test plan:

Change your database structure to add the IsbnOrIssn table:

DROP TABLE IF EXISTS `_DATABASE_NAME_`.`IsbnOrIssn`;
CREATE TABLE  `_DATABASE_NAME_`.`IsbnOrIssn` (
 `isbnOrIssnID` int(11) NOT NULL auto_increment,
 `resourceID` int(11) default NULL,
 `isbnOrIssn` varchar(45) NOT NULL,
 PRIMARY KEY  (`isbnOrIssnID`),
 KEY `resourceID` (`resourceID`),
 KEY `isbnOrIssn` (`isbnOrIssn`)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=latin1;

Remove the IsbnOrIssn field from the Resource table :

ALTER TABLE `Resource` DROP `isbnOrISSN`;

Change the admin/configuration.ini file to specify on which input columns should the IsbnOrIssn be deduped for the import tool:

[settings] importISBNDedupingColumns="ISSN,EISSN,PrintISSN,OnlineISSN,Print ISSN,Online ISSN,ISBN,EISBN"

CSV Example

titleText,resourceURL,ISSN,EISSN,organization,role,parentResource
Accounts of chemical research,http://domain.tld/journal/achre4,0001-4842,1520-4898,ACS,Provider,parent1
Analytical chemistry,http://domain.tld/journal/ancham,0003-2700,1520-6882,ACS2,Consortium,parent1
Biochemistry (ACS),http://domain.tld/journal/bichaw,0006-2960,1520-4995,ACS3,Library,parent2
Bioconjugate chemistry,http://domain.tld/journal/bcches,1043-1802,1520-4812,Audio Engineering Society Inc,Provider,parent2
Chemical research in toxicology,http://domain.tld/journal/crtoec,0893-228X,1520-5010,SRDS Media Solutions,,
Chemical reviews,http://domain.tld/journal/chreay,0009-2665,1520-6890,ACS,,
Chemistry of materials,http://domain.tld/journal/cmatex,0897-4756,1520-5002,ACS,,
Energy & fuels,http://domain.tld/journal/enfuem,0887-0624,1520-5029,,,
Environmental Science & Technology,http://domain.tld/journal/esthag,0013-936X,1520-5851,ACS,,parent2
Environmental Science & Technology,http://domain.tld/journal/esthag,0013-936X,1520-5851,ACS2,,parent1

[PaulPoulain]

Testing this one:

• _DATABASE_NAME_ in the DB update is useless & make the update not work straight
• the ALTER TABLE Resource DROP isbnOrISSN; remove all existing isbn or issn ! yikes ! Please add something like INSERT INTO IsbnOrIssn (SELECT resourceID,resourceID,isbnOrISSN FROM Resource); before the drop isbnOrISSN is much better
• a strange minor display problem: the 2nd parent is not properly aligned: (display on firefox)
• on comment for CSV upload, worked fine. Some minor UI improvement could be done, but this part will not be used frequently, so I think it's OK

Please fix #1, #2 and #3 and we can merge. Also remember to provide a separate sql file for sql update, that will be merged in the global update file (see coding guidelines on the wiki I just created)

[veggiematts]

Update and display fixed !

[remocrevo]

Could you explain a use case for wanting a Resource to have more than one parent Resource?

[veggiematts]

For our customer, a parent Resource is a collection of Resources. A given Resource can belong to several collections (ie: have multiple parents), especially when you know that some resources are mandatory in some collections.

[remocrevo]

Has anyone tested this code? It contains substantial changes, so I think it requires two people testing it a moderate amount, or one institution using it in production for a few days.

[benheet]

Paul P, can you say a bit more about the multiple parent resources. I still do not understand the use case for when multiple parents are required. An example of a specific package or title would help in explaining the need for the feature.

I'm sold on the import, multiple issn/isbn, and deletion of children. Those all look great and I don't think require additional discussion as long as the testing is done that Remington recommended.

[veggiematts]

Benheet, have you seen my comment about multiple parent resources (in response to remocrevo) ?

"For our customer, a parent Resource is a collection of Resources. A given Resource can belong to several collections (ie: have multiple parents), especially when you know that some resources are mandatory in some collections."

I can put this in another terms: when a given resource is mandatory in several collections, having multiple parents is necessary to avoid duplicate resources.

Is that explanation okay or should I go more into details ?

[veggiematts]

Here are some specific examples :

• More than 15 resources are in common between ACS Legacy Archives and ACS Couperin
• 7 resources are in common between Nature ISTEX and Nature Package
• Among others, Journal of Pharmacy & pharmacology belongs to both STM Collection de Wiley and bouquet ISTEX

[benheet]

@veggiematts Thank you for the update. I understand now what you are trying to accomplish. The same journal is in multiple packages and you wish to have one entry in the system for that journal showing its relationship to all parent packages. That is not the data model we used in the original design but I can understand the use case.

Does anyone see any negative impact or implications from this proposed change? Parent/Child relationships are shown within the Product tab and the Helpful Links section so I'd request that you account for any changes to spacing, etc that need to happen to maintain a quality display. Does the Parent/Child relationship have any functionality or display impact anywhere else? Nothing is coming to my mind.

[PaulPoulain]

Le 31/03/2015 20:08, Benjamin Heet a écrit :

@veggiematts https://github.com/veggiematts Thank you for the update. I understand now what you are trying to accomplish. The same journal is in multiple packages and you wish to have one entry in the system for that journal showing its relationship to all parent packages. That is not the data model we used in the original design but I can understand the use case.

Does anyone see any negative impact or implications from this proposed change? Parent/Child relationships are shown within the Product tab and the Helpful Links section so I'd request that you account for any changes to spacing, etc that need to happen to maintain a quality display. Does the Parent/Child relationship have any functionality or display impact anywhere else? Nothing is coming to my mind.

I feel that, for gokb connection, it will be an important feature: gokb has groups of subscriptions, and we should be able to use this multiple parent/child thing !

(and, breaking news: we should have another intern to work with Matts on gokb<>coral, this summer - jun>aug -)

Paul Poulain, Associé-gérant / co-owner BibLibre, expert du logiciel libre pour les bibliothèques BibLibre, Open Source software for libraries expert

[benheet]

i have no objection to the change. if nobody else does then we will just need someone to test and confirm there were no problems.

gokb-coral integration is an exciting opportunity. hopefully we can share thoughts on how that functionality should work at an early stage of development.

[techsvcslib]

One comment I'd make about the multiple parents scenario...or rather, a question. If the same title existed in a JSTOR package (i.e. archival) and also in another package (e.g. for current), does that scenario work for this functionality? If so, I think I'm for it. At this point for these situations, we are creating two separate resource records for the same title. If I'm missing something obvious in all this, help me out.

Steve

[techsvcslib]

I have the answer to my previous question, so please disregard.

[benheet]

@veggiematts have you tested this somewhere other than on your original development install? This has been sitting for a while and we've not had a second tester. If you or Paul can test on a fresh install then we could proceed with the merge based on the result of that testing.

[jeffnm]

We've just installed and tested this. We didn't have the mbstrings php extension installed, but once that and the usual permission fixes and database updates got done, it seems to be working.

The only minor issue we are seeing is that imported records get duplicated rather than overlayed when a new import is run of the same record with some new information. I hear that it is still a good improvement though.

On Fri, Apr 10, 2015 at 11:56 AM Benjamin Heet notifications@github.com wrote:

@veggiematts https://github.com/veggiematts have you tested this somewhere other than on your original development install? This has been sitting for a while and we've not had a second tester. If you or Paul can test on a fresh install then we could proceed with the merge based on the result of that testing.

— Reply to this email directly or view it on GitHub https://github.com/ndlibersa/resources/pull/35#issuecomment-91620951.

[benheet]

@veggiematts can you comment please on the duplication vs overlay? Was the duplication your intention? If it was not intentional, would you rather alter this now or submit this as a github issue to work on later?

[jeffnm]

I'm sorry, I was incorrect. The deduping is working correctly, but there is no overlay of new data into existing records.

[benheet]

That's good to hear. Perhaps this can then go through and we can discuss overlay along with other questions about functionality that Steve started over email.

[jeffnm]

I did not do a fresh install. Does that need to be done still? We've had it in production for one day so far.

[benheet]

My opinion is that the testing you've already done is sufficient and we should go ahead and finish this, especially now that we know it doesn't cause duplicates. This way it can get into the 1.3 version.

[jeffnm]

OK. Go for it.

[remocrevo]

@jeffnm Could you clarify what you mean that "there is no overlay of new data into existing records"? Does that mean some data from the import file should be added to the matching resource but isn't? If that is the expected behavior, it should be clearly documented (e.g. "This tool only supports the following data columns..."). Otherwise it could be misleading.

[veggiematts]

I guess it means is that the import tool only create a resource if it do not already exist. An already existing resource will not be updated or re-created when imported again. Is that what you meant, @jeffnm ?

Anyway, this is an intended behavior.

Updating existing resources through the import tool might be a feature to add in the future.

[jeffnm]

@veggiematts that is correct.

[remocrevo]

I just merged pull request #30 (Cost History Enhancements) into master, then tried merging this pull request with those changes (i.e. the new master branch), and it will require some work. The main challenge is that #30 included splitting the large ajax files into smaller files, and this pull request changes the original ajax files. @veggiematts could you try to resolve the conflicts?

[veggiematts]

Conflicts fixed.