sedimentation-fault
commented
7 years ago I have some new information for you:
First, this bug looks like a duplicate of
https://github.com/ContentMine/getpapers/issues/145
At least, as far as I can see, the symptoms are the same: corrupt PDFs.
Second, I have done some analysis of the contents of the 'good' and 'bad' PDFs of my original post above. The PDFs differ on characters outside the ASCII character set. This is an encoding issue and is known under the name of Mojibake (pronounced "moji-bah-keh"), see:
https://en.wikipedia.org/wiki/Mojibake#English
I say this because, for example, the "standard garbage bytes" that have to be near the beginning of each PDF according to the PDF specs, namely:
%âãÏÓ
appear correctly in the 'good' PDF, but are shown as:
%����
in the 'bad' one. The same is true for lines like
(±)Tj
in the 'good' PDF that appear as
(�)Tj
in the 'bad' one.
I suspect that some tool inside the ContentMine toolchain enforces an 'unnatural' encoding. This encoding transforms non-ASCII characters and we get corruption.
Theoretically, if we knew what encoding is imposed by ContentMine and what is the original one, we could pass the encodings to iconv and transform the PDFs back to original. For example, if we knew that getpapers enforces "codepage 850" to a file that is actually UTF-8, then we could do
iconv -f CP850 -t UTF-8 corrupt.pdf > utf-8.pdf
and utf-8.pdf would be the original one (at least so I think). I have tried this with the following script:
#! /bin/bash
# Accepts two arguments:
#
# - the filename of the PDF with the wrong encoding
# - the filename of the PDF with the correct encoding
#
# Assuming that the correct encoding is UTF-8, the script
# tries all valid iconv encodings and compares the MD5 sums.
# If a MD5 sum matches the MD5 of the PDF with the correct
# encoding, the script reports the source encoding.
# The purpose of this exercise in futility is to find out
# which encoding caused corruption in file_to_test.
file_to_test="$1"
md5_correct="$(md5sum "$2" | awk '{print $1}')"
b="$(basename "$file_to_test" .pdf)"
for a in $(iconv --list | sed 's/\/\/$//' | sort); do
echo -n -e "$a\t"
a_sanitized="$(echo "$a" | sed -e 's/[^0-9a-zA-Z]/-/g; s/--*/-/g')"
iconv -f $a -t UTF-8 "$file_to_test" > "${b}-$a_sanitized.pdf" 2>&1
if [ -e "${b}-$a_sanitized.pdf" ]; then
md5="$(md5sum "${b}-$a_sanitized.pdf" | awk '{print $1}')"
echo -e "$md5\t$md5_correct"
if [ "$md5" = "$md5_correct" ]; then
echo "### FOUND THE CULPRIT! IT IS ENCODING $a !"
break
else
rm "${b}-$a_sanitized.pdf"
fi
fi
donebut it did not find anything...I think I'm close, but some detail escapes me...
petermr
blahah
tarrow
I am totally new to getpapers, so this may be just a misunderstanding on my part of some of the inner workings of the ContentMine toolchain (even though it seems to be pretty straightforward...).
Here is the problem: when I use getpapers to download a paper, say with the command:
getpapers --query 'PMCID:"PMC5293196" JOURNAL:"PLOS ONE"' --outdir network-analysis -pI get:
which looks pretty good to me (the warnings seem to be harmless in my case). Everything is there and the PDF seems to be in
PMC5293196/fulltext.pdfHowever, when I try to open it in acroread, the reader tells me that the file is corrupt, that some font is missing etc. - and the paper is practically unreadable!
If I try to download it directly from the URL
http://europepmc.org/articles/PMC5293196?pdf=render
which is found in the _eupmcresults.json as the download URL for this ID, all is OK and the paper is displayed with absolutely no errors...
There is indeed a difference in the MD5 sums of the two versions:
The latter (PMC5293196.pdf) is the 'good' one, downloaded directly from http://europepmc.org/articles/PMC5293196?pdf=render with curl, the former (PMC5293196/fulltext.pdf) is the 'bad' one downloaded by getpapers.
It is not a locale issue (seems to me), because the same happens to a user with
and to a user with a non-US locale.
I have nodejs 4.4.6 and getpapers is installed with
What is going on here? Can you please shed some light?