PDF preview not working consistently in production

[Mbrownshoes]

This has been more clearly described in #470

See TESTCASE below further in this issue.

We are unable to preview this draft record's pdf resource (results in sever error) https://catalogue.data.gov.bc.ca/dataset/test-geo-1-2/resource/bf5e743a-4320-40dc-8a68-7bdbb11bf064 however, the download does work. This resource is uploaded to the catalogue, though pdfs linked to have experienced the same problem. We also experienced this problem with https://catalogue.data.gov.bc.ca/dataset/caribou-habitat-model-for-the-eastern-cariboo-region-columbia-highlands-northern-columbia-mountains, so I moved the pdf to the description.

Other pdf previews do work, such as this (which is a resource in the same record as above) https://catalogue.data.gov.bc.ca/dataset/test-geo-1-2/resource/1e928ab5-cdd9-4769-8be7-c84ef6944e5c

This behaviour is inconsistent, and I was able to see pdf previews in cat in cad for newly (and old) created resources.

@ll911 suggests solr might need to be upgraded to fix. @dkelsey

[garrettH3S]

@ll911 This error seems to be caused when the resource is coming from a proxy. pdf resource https://catalogue.data.gov.bc.ca/dataset/caribou-habitat-model-for-the-western-cariboo-region-2001/resource/b8809199-96e9-4415-b8b0-899eb8411118/proxy

[ll911]

this is defined by ckan.resource_proxy.max_file_size, default is 1MB, changing this parm will impact performance, need some justification if need to change the value.

[garrettH3S]

@dkelsey If this is an enhancement, what am i enhancing it to? Is this now a duplicate of #372

[dkelsey]

@garrettH3S It's related yes. The behavior I would expect would be if the PDF is larger than 1048576 bytes no preview is displayed.

[dkelsey]

my comments are wrong. The issues is described correctly blow in the TESTCASE and in #470

I created a dataset to verify this issue:

• https://cad.data.gov.bc.ca/dataset/test-large-pdf
• https://cat.data.gov.bc.ca/dataset/test-large-pdf

---

@ll911 :

• ckan.resource_proxy.max_file_size is not set in CAD.
• The docs claim that when not set the default limit is 1MB

@ll911 should ckan.resource_proxy.max_file_size be configure in CAD?

I looked at the current behavior in PROD and forcing a PDF to upload to the DataStore gets the same pdftables in not installed as listed below. I take this to mean we do no store PDFs in the DataStore. PDFs are saved only to the FileStore

---

• I created a test case to validate this issue: test-large-pdf
  • it has 2 resources one is 1.8MB the other is 18.5MB
  • The preview works for both of these. It should not
  • it seems the files are loaded into the FileStore but not the DataStore. When I force and upload for each of these I get two different errors:

1.8 MB file:

This is the behavior is consistent in in CAT.

Error: File "/apps/ckan/tst/datapusher/lib/python2.7/site-packages/apscheduler/scheduler.py", line 512, 
in _run_job retval = job.func(*job.args, **job.kwargs) File "/apps/cis/workspace/bcdc/bcdc-
rc/src/datapusher/datapusher/jobs.py", line 404, in push_to_datastore table_set = 
messytables.any_tableset(tmp, mimetype=ct, extension=ct) File 
"/apps/ckan/tst/datapusher/lib/python2.7/site-packages/messytables/any.py", line 137, in any_tableset 
return parsers[attempt](fileobj, **kw) File "/apps/ckan/tst/datapusher/lib/python2.7/site-
packages/messytables/pdf.py", line 50, in __init__ raise ImportError("pdftables is not installed") 
ImportError('pdftables is not installed',)

In CAD there is another error--

'Could not connect to DataPusher'

18.5 MB File:

This is the behavior in CAT.

Error: Resource too large to process: 10492139 > max (10485760).

In CAD there is another error.

• There is a notice that the file is too large to upload.
• it seems to partially upload and create the resource, though causes other errors.

[jeff-at-h3]

@dkelsey this issue is known to be not working. Dave, do you want us to continue working on it, or do you want to do more testing first?

[dkelsey]

@jeff-at-h3 I realized a while ago that I had the sizes wrong. the max size in 150 MB
I'm testing this today.

[dkelsey]

This is a known issue...not a new one. Lets asses priorities. Hold off working on this for now.

[dkelsey]

This has now been more clearly described in #470

figured out the test case

ENV

CAD, CAT, PROD

TESTCASE

• Edit a record and add a resource
• Instead of 'Upload' use 'Link'
• use the following URL for the link: http://www.env.gov.bc.ca/cariboo/env_stewardship/wildlife/inventory/caribou/northcar/hmi/hsi06-01.pdf
• complete remaining mandatory fields
• submit to add the resource
• navaigate the the resource page for the newly created resource.

  EXPECTED

• The preview either displays the PDF or the preview does not show at all because the file is too large.

  ACTUAL