search

PresConsUIUC / PSAP

Home of the Preservation Self-Assessment Program application.
https://psap.library.illinois.edu/
Other
4 stars 1 forks source link

Institution CSV export takes forever #205

Closed adolski closed 9 years ago

adolski commented 9 years ago

Feedback requested on this:

Currently, the multiple-resource CSV export contains hundreds of columns: 20 or so for each property of the resources, plus one for each assessment question in the system, of which there are several hundred. This is necessary because different resources will have responses to different questions.

For technical reasons, generating this file is super slow. I didn't realize how slow it would be when I wrote the code way back in the summer, before we had so many assessment questions. I could potentially modify the code to display a set of only the questions available to any of the resources' formats, but (without explaining, in the interest of keeping this brief) that would be a poor solution.

Presumably the main advantage of CSV is the ability to open it in Excel, but this data doesn't really benefit from Excel's features -- it's not chartable, for example. If the intent is to make data available in a format that can be imported into other systems, any programmer would prefer JSON, which is so much richer.

So, I propose retaining CSV export excluding assessment questions, and adding a JSON export that includes assessment questions. Does this sound OK?

adolski commented 9 years ago

This is in place now; if it ends up being a problem, we can revisit it then.

jhain commented 9 years ago

That works for me. we are also exporting in XML, or no longer?

J

From: adolski [mailto:notifications@github.com] Sent: Friday, January 09, 2015 12:20 PM To: PresConsUIUC/PSAP Subject: Re: [PSAP] Institution CSV export takes forever (#205)

Feedback requested on this:

Currently, the multiple-resource CSV export contains hundreds of columns: 20 or so for each property of the resources, plus one for each assessment question in the system, of which there are several hundred. This is necessary because different resources will have responses to different questions.

For technical reasons, generating this file is super slow. I could potentially modify the code to display a set of only the questions available to any of the resources' formats, but (without explaining, in the interest of keeping this brief) that would be a poor solution.

Presumably the main advantage of CSV is the ability to open it in Excel, but this data doesn't really benefit from Excel's features -- it's not chartable, for example. If the intent is to make data available in a format that can be imported into other systems, any programmer would prefer JSON, which is so much richer.

So, I propose retaining CSV export excluding assessment questions, and adding a JSON export that includes assessment questions. Does this sound OK?

— Reply to this email directly or view it on GitHubhttps://github.com/PresConsUIUC/PSAP/issues/205#issuecomment-69375188.

adolski commented 9 years ago

@jhain There is an ability to export individual resources in EAD XML format, but not multiple resources, for whatever reason. I could easily add that, though. I can demo how all of this exporting works at the meeting tomorrow.

jhain commented 9 years ago

I think a demo would help me wrap my head around the differences, though I totally get the cumbersome excel export.

From: adolski [mailto:notifications@github.com] Sent: Tuesday, January 20, 2015 3:25 PM To: PresConsUIUC/PSAP Cc: Teper, Jennifer Hain Subject: Re: [PSAP] Institution CSV export takes forever (#205)

@jhainhttps://github.com/jhain There is an ability to export individual resources in EAD XML format, but not multiple resources, for whatever reason. I could easily add that, though. I can demo how all of this exporting works at the meeting tomorrow.

— Reply to this email directly or view it on GitHubhttps://github.com/PresConsUIUC/PSAP/issues/205#issuecomment-70736606.