infolab-csail / wikipedia-mirror

Makefiles that will download and setup a local wikipedia instance.
1 stars 2 forks source link

make sql-load-parts, Syntax error: Unterminated quoted string #16

Closed michaelsilver closed 9 years ago

michaelsilver commented 9 years ago

After a successful make sql-dump-parts, make sql-load-parts fails with the error "Syntax error: Unterminated quoted string" and the output:

root@futuna:/scratch/wikipedia-mirror# make sql-load-parts
Makefile.smartdumps:307: target `/scratch/wikipedia-mirror/drafts/wikipedia-parts/enwiki-20140614-categorylinks.sql-loaded' given more than once in the same rule.
Makefile.smartdumps:307: target `/scratch/wikipedia-mirror/drafts/wikipedia-parts/enwiki-20140614-redirect.sql-loaded' given more than once in the same rule.
Makefile.smartdumps:307: target `/scratch/wikipedia-mirror/drafts/wikipedia-parts/enwiki-20140614-pagelinks.sql-loaded' given more than once in the same rule.
make: Circular bmw-install <- /scratch/wikipedia-mirror/fs/ctlscript.sh dependency dropped.
echo "==== Running locally ====" &&  if [ ! -d /scratch/wikipedia-mirror/sources/mwdumper-git ]; then git clone https://gerrit.wikimedia.org/r/p/mediawiki/tools/mwdumper.git /scratch/wikipedia-mirror/sources/mwdumper-git ; if [ "" != "" ]; then git checkout ; fi; fi
==== Running locally ====
if [ ! -f /scratch/wikipedia-mirror/tools/mwdumper.jar ]; then \
                cd /scratch/wikipedia-mirror/sources/mwdumper-git/ &&  mvn -Duser.home="/scratch/wikipedia-mirror/drafts" package; \
                cp /scratch/wikipedia-mirror/sources/mwdumper-git/target/mwdumper-1.16.jar /scratch/wikipedia-mirror/tools/mwdumper.jar; \
        fi
if [ -e "$(ps aux | grep 'mysq[l]')" ]; then \
                echo "already running mysql"; \
        else \  
                /scratch/wikipedia-mirror/fs/ctlscript.sh start; \
        fi
/scratch/wikipedia-mirror/fs/mysql/scripts/ctl.sh : mysql  (pid 10179) already running
Syntax OK
/scratch/wikipedia-mirror/fs/apache2/scripts/ctl.sh : httpd (pid 10411) already running
loading: /scratch/wikipedia-mirror/drafts/wikipedia-parts/enwiki-20150205-pages-meta-current1.xml-p000000010p000010000.sql...
bash -c '( echo 'SET AUTOCOMMIT = 0; SET FOREIGN_KEY_CHECKS=0;' && cat  /scratch/wikipedia-mirror/drafts/wikipedia-parts/enwiki-20150205-pages-meta-current1.xml-p000000010p000010000.sql && echo 'SET FOREIGN_KEY_CHECKS=1;COMMIT;SET AUTOCOMMIT=1;'; ) | stdbuf -oL tee >( /scratch/wikipedia-mirror/fs/mysql/bin/mysql -u csail --password=pass bitnami_mediawiki --force )' | stdbuf -oL awk '  | \
                stdbuf -o 'L' tee /scratch/wikipedia-mirror/fs/mysql-progress.log 2> /scratch/wikipedia-mirror/fs/mysql-error.log && \
        touch /scratch/wikipedia-mirror/drafts/wikipedia-parts/enwiki-20150205-pages-meta-current1.xml-p000000010p000010000.sql-loaded
/bin/sh: 3: Syntax error: Unterminated quoted string
make: *** [/scratch/wikipedia-mirror/drafts/wikipedia-parts/enwiki-20150205-pages-meta-current1.xml-p000000010p000010000.sql-loaded] Error 2

I know, in general, that this error means a string somewhere is not ended property, for example, with a matching end-quotation mark. Looking at the source of Makefile.smartdumps, I did not find any unpaired quotation marks (as this error usually means); but, I do suspect the problem has to do with the usage of tee in mysql-load-file:

# Dont pass quotes here, this will tee the output. Note that we can't
# get process substitution with POSIX shell so we need bash
mysql-load-file= bash -c "(                         \
        echo 'SET AUTOCOMMIT = 0; SET FOREIGN_KEY_CHECKS=0;' &&     \
        cat $1 &&                           \
        echo 'SET FOREIGN_KEY_CHECKS=1;COMMIT;SET AUTOCOMMIT=1;';   \
    ) | stdbuf -oL tee >( $(MYSQL_CMD) )"

Although I suspect it's a simple error to fix, I was not able to figure it out (I'm a newbie to Bash scripting). Could you please help me out?

fakedrake commented 9 years ago

Ok I fixed it, it was an extra newline on Makefile.smartdumps:222. For the record you also need gawk (instead of oawk(?) that seems to come with ubuntu)