QA New Datastore

[janette]

• [ ] large file
• [ ] historically difficult file
• [ ] recline previews work
• [ ] embeded preview works

[janette]

Done

http://janette.dkandemo.nuamsdev.com/search/type/dataset

I could not find a "No active batch." message anywhere in dkan, that must be coming from sowhere else. Done

http://janette.dkandemo.nuamsdev.com/admin/content/datastore

Done

[janette]

VA resource test http://janette.dkandemo.nuamsdev.com/node/4/datastore

Invalid line 1 in https://www.va.gov/DIGITALSTRATEGY/cemdata/states/ngl_alabama.csv; header: ["d_first_name","d_mid_name","d_last_name","d_suffix","d_birth_date","d_death_date","section_id","row_num","site_num","cem_name","cem_addr_one","cem_addr_two","city","state","zip","cem_url","cem_phone","relationship","v_first_name","v_mid_name","v_last_name","v_suffix","branch","rank","war"] values: ["Winston","Russell","Langley",null,"08\/24\/1950","03\/09\/2018","2",null,"1341","ALABAMA NATIONAL CEMETERY","3133 HIGHWAY 119",null,"MONTEVALLO","AL","35115","http:\/\/www.cem.va.gov\/CEM\/cems\/nchp\/alabama.asp","205-665-9039","Veteran (Self)","Winston","Russell","Langley",null,"US MARINE CORPS","LCPL","VIETNAM",""]

Works when Trailing delimiter box is checked, maybe we could add the suggestion to the status message

[erogray]

When importing the CSV at http://janette.dkandemo.nuamsdev.com/dataset/ero-test-data/resource/3282ee42-50e6-4e5c-8bce-1df00566aa27#{} I got a "SQLSTATE[42000]: Syntax error or access violation: 1118 Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. You have to change some columns to TEXT or BLOBs" error.

[erogray]

Clicking the "Download" button on http://janette.dkandemo.nuamsdev.com/dataset/ero-test-data/resource/3282ee42-50e6-4e5c-8bce-1df00566aa27#{} led me to http://janette.dkandemo.nuamsdev.com/node/9/download, which doesn't seem to load.

Possibly related: after returning to the dataset, I see the following error message:

Done

[erogray]

Trying to import the gravesite data (from a VA resource) for http://janette.dkandemo.nuamsdev.com/dataset/ero-test-data/resource/78694916-ffb0-4b6f-8656-a536e5fe8832#{} yielded a generic Error:

[erogray]

After importing http://janette.dkandemo.nuamsdev.com/dataset/ero-test-data/resource/7549be3e-179b-4db3-8cdb-008475bbf5a1 (which seemed successful) I viewed the API (seems to look good), but then when I hit the Back button to get back to the Resource, I got a formidable block of pink.

Done

[erogray]

If I attempt to import a PDF file, the importer shows some mixed messages.

Probably we'd prefer some sort of "That's not an importable filetype" message?

[erogray]

After importing the standard test fixture http://janette.dkandemo.nuamsdev.com/dataset/ero-test-data/resource/bd2b8c33-4def-4a58-9a7e-f1e8075ed338 and clicking the Download button (which, as previously noted, doesn't work) I get an interesting combination of error messages, depending on which page I look at:

[erogray]

I am pleased, but puzzled, that a really gigantic resource (http://janette.dkandemo.nuamsdev.com/dataset/ero-test-data/resource/5b3e2f8c-7d1f-4505-a81a-4c17f5439c64#{}) is previewable right after import is begun, but before it has completed.

This seems like a big improvement for useability.

[erogray]

Download button seems to work correctly now!

On https://ba15eb9d-e36c-48e4-821f-df693ed795d7--pr-2734.probo.build/node/45/datastore I got some gnarly import errors on what seems like a pretty normal CSV (https://ba15eb9d-e36c-48e4-821f-df693ed795d7--pr-2734.probo.build/dataset/ero-test-data/resource/22c408a4-7b64-448a-b1dd-c10021cb7f22):

[erogray]

When adding a non-CSV file (so far, I've seen it happen on PDFs and JSONs) a message appears saying "Incorrect filemime application/filetype". This is likely to confuse users, who think they should be able to add any allowed filetype.

[janette]

@erogray even with the built-in disclaimer at the top?

[erogray]

When adding a remote CSV, https://ba15eb9d-e36c-48e4-821f-df693ed795d7--pr-2734.probo.build/dataset/test-dataset/resource/978c60a5-12f5-4252-a03c-9782e522b3ce , I got a "Node 44 doesn't have a proper file path." message:

this is potentially problematic because (a) it looks like an error; (b) there's no way to import the remote file to the datastore.

[janette]

@erogray @stefaniefgray The above example resource was added to the API or Website URL field, so would not be expected to be added to the datastore. The user would either need to use the upload field or remote file field in order to add it to the datastore

[erogray]

When adding a remote file that's an XLS, http://www.energy.ca.gov/almanac/electricity_data/vintage/Vintage_Annual_Power_Plant_capacity.xls to https://ba15eb9d-e36c-48e4-821f-df693ed795d7--pr-2734.probo.build/dataset/us-tobacco-usage-statistics/resource/311727ad-a0b8-49c3-bbc8-0e20d0d91682 I got "Notice: Undefined offset: 1 in DrupalRemoteStreamWrapper::getHeaders() (line 96 of /src/dkan/modules/contrib/remote_stream_wrapper/remote_stream_wrapper.inc)."

[erogray]

As with the XLS file linked to above, it seems like in general some filetypes are not correctly icon-identified when a resource is created. A Zip file also shows a "data" icon in https://ba15eb9d-e36c-48e4-821f-df693ed795d7--pr-2734.probo.build/dataset/us-tobacco-usage-statistics/resource/ba83ef35-f0d2-422d-9977-49f13d58291c

A moment later, however, I see that the Excel resource is now correctly identified with an Excel icon, and an error appears on the Zip resource: "Notice: Trying to get property of non-object in Dkan\Datastore\Resource::filePath() (line 116 of /src/dkan/modules/dkan/dkan_datastore/src/Resource.php)."

[erogray]

I've added a TSV file: https://ba15eb9d-e36c-48e4-821f-df693ed795d7--pr-2734.probo.build/dataset/florida-bike-lanes/resource/8599ca38-8fd1-4590-a2ba-0babd693569f#{} but I get a "cannot be imported to our datastore." message, which seems incorrect for this filetype.

[erogray]

QA is complete, moving to test on actual sites