search

HazyResearch / deepdive

DeepDive
deepdive.stanford.edu
1.95k stars 539 forks source link

Spouse example corenlp: invalid COMMAND #604

Open profressor opened 7 years ago

profressor commented 7 years ago

Hi, following your spouse tutorial, I managed to fix some issues on they way that popped up following exactly the instructions. However now Im stucked. Im at the spouse example (step "deepdive do sentences"). Using latest stable Deepdive, Postgresql, [l]Ubuntu 16.04.1

user@vm:~/spouse$ deepdive do sentences ‘run/RUNNING’ -> ‘20161115/212239.785459092’ ‘run/LATEST’ -> ‘20161115/212239.785459092’ 2016-11-15 21:22:44.268156 # on vm: deepdive do sentences 2016-11-15 21:22:44.268231 # run/20161115/212239.785459092/plan.sh 2016-11-15 21:22:44.268244 # execution plan for data/sentences 2016-11-15 21:22:44.268250 2016-11-15 21:22:44.268256 : ## process/init/app ########################################################## 2016-11-15 21:22:44.268261 : # Done: 2016-11-15T21:10:39+0100 (12m ago) 2016-11-15 21:22:44.268266 : process/init/app/run.sh 2016-11-15 21:22:44.268271 : mark_done process/init/app 2016-11-15 21:22:44.268278 : ############################################################################## 2016-11-15 21:22:44.268283 2016-11-15 21:22:44.268287 2016-11-15 21:22:44.268292 : ## process/init/relation/articles ############################################ 2016-11-15 21:22:44.268297 : # Done: 2016-11-15T21:10:39+0100 (12m ago) 2016-11-15 21:22:44.268308 : process/init/relation/articles/run.sh 2016-11-15 21:22:44.268314 : mark_done process/init/relation/articles 2016-11-15 21:22:44.268318 : ############################################################################## 2016-11-15 21:22:44.268323 2016-11-15 21:22:44.268328 2016-11-15 21:22:44.268332 : ## data/articles ############################################################# 2016-11-15 21:22:44.268339 : # Done: 2016-11-15T21:10:39+0100 (12m ago) 2016-11-15 21:22:44.268344 : # no-op 2016-11-15 21:22:44.268349 : mark_done data/articles 2016-11-15 21:22:44.268353 : ############################################################################## 2016-11-15 21:22:44.268358 2016-11-15 21:22:44.268363 2016-11-15 21:22:44.268370 ## process/ext_sentences_by_nlp_markup ####################################### 2016-11-15 21:22:44.268375 # Done: N/A 2016-11-15 21:22:44.268380 process/ext_sentences_by_nlp_markup/run.sh 2016-11-15 21:22:44.268384 ++ dirname process/ext_sentences_by_nlp_markup/run.sh 2016-11-15 21:22:44.268389 + cd process/ext_sentences_by_nlp_markup 2016-11-15 21:22:44.268394 + export DEEPDIVE_CURRENT_PROCESS_NAME=process/ext_sentences_by_nlp_markup 2016-11-15 21:22:44.268400 + DEEPDIVE_CURRENT_PROCESS_NAME=process/ext_sentences_by_nlp_markup 2016-11-15 21:22:44.268405 + export DEEPDIVE_LOAD_FORMAT=tsv 2016-11-15 21:22:44.268410 + DEEPDIVE_LOAD_FORMAT=tsv 2016-11-15 21:22:44.268415 + deepdive compute execute 'input_sql= SELECT R0.id AS "articles.R0.id", R0.content AS "articles.R0.content" 2016-11-15 21:22:44.268419 FROM articles R0 2016-11-15 21:22:44.268424
2016-11-15 21:22:44.268429 ' 'command="$DEEPDIVE_APP"/udf/nlp_markup.sh' output_relation=sentences 2016-11-15 21:22:44.275046 Executing with the following configuration: 2016-11-15 21:22:44.275082 DEEPDIVE_NUM_PROCESSES=3 2016-11-15 21:22:44.275092 DEEPDIVE_NUM_PARALLEL_UNLOADS=1 2016-11-15 21:22:44.275099 DEEPDIVE_NUM_PARALLEL_LOADS=1 2016-11-15 21:22:44.275143 output_relation_tmp=dd_tmp_sentences 2016-11-15 21:22:44.275160 2016-11-15 21:22:44.377662 CREATE TABLE 2016-11-15 21:22:44.539507 CREATE TABLE 2016-11-15 21:22:44.573678 unloading to feed_processes-1: ' SELECT R0.id AS "articles.R0.id", R0.content AS "articles.R0.content" 2016-11-15 21:22:44.573751 FROM articles R0 2016-11-15 21:22:44.573760
2016-11-15 21:22:44.573765 ' 2016-11-15 21:22:44.592157 Loading dd_tmp_sentences from output_computed-1 (tsv format) > 2016-11-15 21:22:44.592359 [ERROR] corenlp: invalid COMMAND 2016-11-15 21:22:44.595204 [ERROR] corenlp: invalid COMMAND 2016-11-15 21:22:44.595754 [ERROR] corenlp: invalid COMMAND 2016-11-15 21:22:44.604159 [ERROR] corenlp: invalid COMMAND 2016-11-15 21:22:44.604910 PLEASE MAKE SURE YOU HAVE RUN: deepdive corenlp start 2016-11-15 21:22:44.607248 [ERROR] corenlp: invalid COMMAND 2016-11-15 21:22:44.608122 PLEASE MAKE SURE YOU HAVE RUN: deepdive corenlp start 2016-11-15 21:22:44.609329 [ERROR] corenlp: invalid COMMAND 2016-11-15 21:22:44.609867 PLEASE MAKE SURE YOU HAVE RUN: deepdive corenlp start loading dd_tmp_sentences: 0:00:00 582 B [46,6KiB/s] ([46,6KiB/s])cal/compute-execute: line 140: kill: (7400) - No such process 2016-11-15 21:22:44.610669 /home/user/local/util/compute-driver/local/compute-execute: line 140: kill: (7401) - No such process loading dd_tmp_sentences: 0:00:00 18 [1,47k/s] ([1,47k/s])er/local/compute-execute: line 140: kill: (7402) - No such process 2016-11-15 21:22:44.612347 [ERROR] command='"$DEEPDIVE_APP"/udf/nlp_markup.sh': PID 7400: finished with non-zero exit status (0) 2016-11-15 21:22:44.612440 /home/user/local/util/compute-driver/local/compute-execute: line 138: 7407 Terminated deepdive-load "$output_relation_tmp" output_computed- 2016-11-15 21:22:44.612458 /home/user/local/util/compute-driver/local/compute-execute: line 138: 7414 Terminated mkmimo feed_processes- > punloading: 0:00:00 0 [ 0 /s] ([ 0 /s]) unloading: 0:00:00 12KiB [ 308KiB/s] ([ 308KiB/s]) 2016-11-15 21:22:44.655451 ERROR: missing data for column "sentence_index" 2016-11-15 21:22:44.655532 CONTEXT: COPY dd_tmp_sentences, line 1: "# DeepDive (v0.8.0-79-g28a58de)" ‘run/ABORTED’ -> ‘20161115/212239.785459092’

Any help appreciated. Also:

user@vm:~/spouse$ deepdive corenlp start

DeepDive (v0.8.0-79-g28a58de)

Usage: deepdive COMMAND [ARG]...

To enable bash completion for deepdive commands, run:

source $(deepdive whereis installed etc/deepdive_bash_completion.sh)

corenlp: invalid COMMAND

Running etc/deepdive_bash_completion.sh just does nothing. If that helps.

http://pastebin.com/uxQ7z7fM

netj commented 7 years ago

Seems like you're using the example code on the master branch but the 0.8-STABLE version for DeepDive. We'll release 0.9 soon that can run the example on master. Meanwhile, I recommend running our Docker images: https://github.com/HazyResearch/deepdive/blob/master/doc/installation.md#launch-without-installing

profressor commented 7 years ago

Tried the docker images. Didnt work

user@vm:~$ bash <(curl -fsSL git.io/getdeepdive) ++ get-url https://github.com/HazyResearch/deepdive/raw/master/util/install.sh ++ curl -fsSL https://github.com/HazyResearch/deepdive/raw/master/util/install.sh

DeepDive installer for Ubuntu

1) deepdive 6) postgres 2) deepdive_docker_sandbox 7) postgres_xl 3) deepdive_example_notebook 8) run_deepdive_tests 4) deepdive_from_release 9) spouse_example 5) jupyter_notebook

Install what (enter to repeat options, a to see all, q to quit, or a number)? 2

Starting installation for deepdive_docker_sandbox

Is there a way to switch from stable to master build ? Or any other fix for this problem?

When will v0.9 be released?

netj commented 7 years ago

@profressor Sorry but there was a glitch in the installer. It is now fixed, but for extra certainty, you can use the following command:

INSTALLER_BRANCH=master RELEASE=master  bash <(curl -fsSL git.io/getdeepdive) deepdive_docker_sandbox
profressor commented 7 years ago

Using that command gives me another error

Step 5 : USER jovyan ---> Using cache ---> d04688cef943 Step 6 : ADD install.sh /deepdive/ ERROR: Service 'deepdive-notebooks' failed to build: lstat install.sh: no such file or directory

Failed installation for deepdive_docker_sandbox

Any ideas?

netj commented 7 years ago

@profressor Thanks for spotting another glitch. The fix is just pushed now, and I made sure it works for me on a fresh Docker for Mac installation. This Docker sandbox we're introducing in this release, honestly, hasn't been tested thoroughly or used widely yet, so your early feedback is very helpful.

profressor commented 7 years ago

The Docker sandbox installation works now. However Im not familiar how to use it to build and run my own DD apps what I intend to do. Using my prefered method No. 1. (Deepdive installer) however shows some error

✗ partition_integers works (in test file util/test/partition_integers.bats, line 43) `partition_integers $m $n | check_correct_output $m $n' failed partitioning 0 by 2 /tmp/bats.72064.src: line 6: incorrect_lines: Permission denied ✓ partition_integers fails on non-positive partitions

125 tests, 1 failure, 5 skipped

Failed installation for run_deepdive_tests

Looks like a tiny perm. error which can be fixed easily. Back to topic, Im still stuck on the spouse example doing the sentences. Im using the updated version which still gives me (another) error:

user@vm:~/spouse$ deepdive compile (OK)

user@vm:~/spouse$ deepdive do sentences ‘run/RUNNING’ -> ‘20161118/002015.669047933’ ‘run/LATEST’ -> ‘20161118/002015.669047933’ 2016-11-18 00:20:19.019593 # on vm: deepdive do sentences 2016-11-18 00:20:19.019657 # run/20161118/002015.669047933/plan.sh 2016-11-18 00:20:19.019670 # execution plan for data/sentences 2016-11-18 00:20:19.019677 2016-11-18 00:20:19.019686 : ## process/init/app ########################################################## 2016-11-18 00:20:19.019693 : # Done: 2016-11-18T00:06:32+0100 (13m 43s ago) 2016-11-18 00:20:19.019699 : process/init/app/run.sh 2016-11-18 00:20:19.019705 : mark_done process/init/app 2016-11-18 00:20:19.019712 : ############################################################################## 2016-11-18 00:20:19.019721 2016-11-18 00:20:19.019727 2016-11-18 00:20:19.019733 : ## process/init/relation/articles ############################################ 2016-11-18 00:20:19.019739 : # Done: 2016-11-18T00:06:32+0100 (13m 43s ago) 2016-11-18 00:20:19.019745 : process/init/relation/articles/run.sh 2016-11-18 00:20:19.019750 : mark_done process/init/relation/articles 2016-11-18 00:20:19.019756 : ############################################################################## 2016-11-18 00:20:19.019764 2016-11-18 00:20:19.019770 2016-11-18 00:20:19.019776 : ## data/articles ############################################################# 2016-11-18 00:20:19.019782 : # Done: 2016-11-18T00:06:32+0100 (13m 43s ago) 2016-11-18 00:20:19.019787 : # no-op 2016-11-18 00:20:19.019793 : mark_done data/articles 2016-11-18 00:20:19.019800 : ############################################################################## 2016-11-18 00:20:19.019807 2016-11-18 00:20:19.019812 2016-11-18 00:20:19.019818 ## process/ext_sentences_by_nlp_markup ####################################### 2016-11-18 00:20:19.019825 # Done: N/A 2016-11-18 00:20:19.019831 process/ext_sentences_by_nlp_markup/run.sh 2016-11-18 00:20:19.019837 ++ dirname process/ext_sentences_by_nlp_markup/run.sh 2016-11-18 00:20:19.019844 + cd process/ext_sentences_by_nlp_markup 2016-11-18 00:20:19.019850 + export DEEPDIVE_CURRENT_PROCESS_NAME=process/ext_sentences_by_nlp_markup 2016-11-18 00:20:19.019856 + DEEPDIVE_CURRENT_PROCESS_NAME=process/ext_sentences_by_nlp_markup 2016-11-18 00:20:19.019862 + export DEEPDIVE_LOAD_FORMAT=tsv 2016-11-18 00:20:19.019869 + DEEPDIVE_LOAD_FORMAT=tsv 2016-11-18 00:20:19.019875 + deepdive compute execute 'input_sql= SELECT R0.id AS "articles.R0.id", R0.content AS "articles.R0.content" 2016-11-18 00:20:19.019883 FROM articles R0 2016-11-18 00:20:19.019889
2016-11-18 00:20:19.019894 ' 'command="$DEEPDIVE_APP"/udf/nlp_markup.sh' output_relation=sentences 2016-11-18 00:20:19.027645 Executing with the following configuration: 2016-11-18 00:20:19.027708 DEEPDIVE_NUM_PROCESSES=3 2016-11-18 00:20:19.027720 DEEPDIVE_NUM_PARALLEL_UNLOADS=1 2016-11-18 00:20:19.027727 DEEPDIVE_NUM_PARALLEL_LOADS=1 2016-11-18 00:20:19.027734 output_relation_tmp=dd_tmp_sentences 2016-11-18 00:20:19.027739 2016-11-18 00:20:19.116391 CREATE TABLE 2016-11-18 00:20:19.237686 CREATE TABLE 2016-11-18 00:20:19.275085 unloading to feed_processes-1: ' SELECT R0.id AS "articles.R0.id", R0.content AS "articles.R0.content" 2016-11-18 00:20:19.275149 FROM articles R0 2016-11-18 00:20:19.275160
2016-11-18 00:20:19.275168 ' > 2016-11-18 00:20:19.278372 No Bazaar/Parser set up at: /home/user/spouse/udf/bazaar/parser 2016-11-18 00:20:19.278409 /home/user/local/util/compute-driver/local/compute-execute: line 140: kill: (5382) - No such process 2016-11-18 00:20:19.279831 [ERROR] command='"$DEEPDIVE_APP"/udf/nlp_markup.sh': PID 5382: finished with non-zero exit status (0) 2016-11-18 00:20:19.279923 /home/user/local/util/compute-driver/local/compute-execute: line 138: 5383 Terminated DEEPDIVE_CURRENT_PROCESS_INDEX=$i bash -c "$command" < process-$i.input > process-$i.output 2016-11-18 00:20:19.279942 /home/user/local/util/compute-driver/local/compute-execute: line 138: 5384 Terminated DEEPDIVE_CURRENT_PROCESSINDEX=$i bash -c "$command" < process-$i.input > process-$i.output 2016-11-18 00:20:19.279949 /home/user/local/util/compute-driver/local/compute-execute: line 138: 5388 Terminated mkmimo process-.output > outputcomputed- 2016-11-18 00:20:19.279971 /home/user/local/util/compute-driver/local/compute-execute: line 138: 5389 Terminated deepdive-load "$output_relation_tmp" output_computed- 2016-11-18 00:20:19.279984 /home/user/local/util/compute-driver/local/compute-execute: line 138: 5395 Terminated deepdive-db unload "$input_sql" "$DEEPDIVE_LOAD_FORMAT" feed_processes- unloading: 0:00:00 0 [ 0 /s]([ 0 /s])il/compute-driver/local/compute-execute: line 138: 5396 Terminated mkmimo feed_processes-* > process-*.input unloading: 0:00:00 12KiB [ 546KiB/s]([ 546KiB/s]) ‘run/ABORTED’ -> ‘20161118/002015.669047933’

Can this be fixed? I need to run the spouse example in order to understand how DD works. I need to get it working to finish my own research on my masters thesis about DeepDive. Any help appreciated!

netj commented 7 years ago

You should run the spouse example within the sandbox. The output you posted seems to be coming from the deepdive 0.8.x installed outside the sandbox. The sandbox includes the spouse example code and data with new version of deepdive that no longer relies on bazaar/parser. It should be very handy to try and apply to your own data.

profressor commented 7 years ago

Hi Jaeho,

Im using the sandbox image now but there is a problem connecting to the database. Im not too familiar with docker :/

bash <(curl -fsSL git.io/getdeepdive)
...
deepdive-notebooks_1  | [W 14:10:01.971 NotebookApp] Widgets are unavailable. Please install widgetsnbextension or ipywidgets 4.0
deepdive-notebooks_1  | [I 14:10:01.990 NotebookApp] Serving notebooks from local directory: /ConfinedWater
deepdive-notebooks_1  | [I 14:10:01.990 NotebookApp] 0 active kernels 
deepdive-notebooks_1  | [I 14:10:01.990 NotebookApp] The Jupyter Notebook is running at: http://0.0.0.0:8888/
deepdive-notebooks_1  | [I 14:10:01.990 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

user@vm:~$ docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS               NAMES
dc7ee2c58466        hazyresearch/postgres   "/docker-entrypoint.s"   4 days ago          Up 24 minutes       5432/tcp            sandbox_database_1

user@vm:~$ docker run -i -t -v $HOME:/vm hazyresearch/deepdive:latest /bin/bash

jovyan@6353d26f2fd8:/vm/spouse$ ls
app.ddlog  db.url  deepdive.conf  input  run  udf

jovyan@6353d26f2fd8:/vm/spouse$ deepdive compile
‘run/compiled’ -> ‘20161122/143702.284636268’

jovyan@6353d26f2fd8:/vm/spouse$ deepdive do sentences
‘run/RUNNING’ -> ‘20161122/143608.888898717’
2016-11-22 14:36:09.071365 process/ext_sentences_by_nlp_markup/run.sh
2016-11-22 14:36:09.071107 ################################################################################
2016-11-22 14:36:09.071216 # Host: 6353d26f2fd8
2016-11-22 14:36:09.071236 # DeepDive: deepdive v0.8.0-742-g4b812a1 (Linux x86_64)
2016-11-22 14:36:09.071247 export PATH='/usr/local/bin':"$PATH"
2016-11-22 14:36:09.071256 export DEEPDIVE_PWD='/vm/spouse'
2016-11-22 14:36:09.071265 export DEEPDIVE_APP='/vm/spouse'
2016-11-22 14:36:09.071278 cd "$DEEPDIVE_APP"/run
2016-11-22 14:36:09.071288 export DEEPDIVE_RUN_ID='20161122/143608.888898717'
2016-11-22 14:36:09.071297 # Plan: 20161122/143608.888898717/plan.sh
2016-11-22 14:36:09.071306 # Targets: sentences
2016-11-22 14:36:09.071315 ################################################################################
2016-11-22 14:36:09.071324 
2016-11-22 14:36:09.071334     # process/init/app/run.sh ####################################### last done: 2016-11-21T23:38:35+0000 (14h 57m 33s ago)
2016-11-22 14:36:09.071344     # process/init/relation/articles/run.sh ######################### last done: 2016-11-21T23:38:35+0000 (14h 57m 33s ago)
2016-11-22 14:36:09.071353     # deepdive mark 'done' data/articles ############################ last done: 2016-11-21T23:38:35+0000 (14h 57m 33s ago)
2016-11-22 14:36:09.071365 process/ext_sentences_by_nlp_markup/run.sh ########################## last done: N/A
2016-11-22 14:36:09.071375 ++ dirname process/ext_sentences_by_nlp_markup/run.sh
2016-11-22 14:36:09.071384 + cd process/ext_sentences_by_nlp_markup
2016-11-22 14:36:09.071407 + : dd_tmp_ dd_old_
2016-11-22 14:36:09.071436 + export DEEPDIVE_CURRENT_PROCESS_NAME=process/ext_sentences_by_nlp_markup
2016-11-22 14:36:09.071467 + DEEPDIVE_CURRENT_PROCESS_NAME=process/ext_sentences_by_nlp_markup
2016-11-22 14:36:09.071487 + export DEEPDIVE_LOAD_FORMAT=tsv
2016-11-22 14:36:09.071524 + DEEPDIVE_LOAD_FORMAT=tsv
2016-11-22 14:36:09.071553 + output_relation=sentences
2016-11-22 14:36:09.071568 + output_relation_tmp=dd_tmp_sentences
2016-11-22 14:36:09.071579 + output_relation_old=dd_old_sentences
2016-11-22 14:36:09.071591 + deepdive create table-if-not-exists sentences
2016-11-22 14:36:09.192254 psql: could not connect to server: Connection refused
2016-11-22 14:36:09.192330  Is the server running on host "localhost" (::1) and accepting
2016-11-22 14:36:09.192347  TCP/IP connections on port 5432?
2016-11-22 14:36:09.192367 could not connect to server: Connection refused
2016-11-22 14:36:09.192377  Is the server running on host "localhost" (127.0.0.1) and accepting
2016-11-22 14:36:09.192390  TCP/IP connections on port 5432?
2016-11-22 14:36:09.227064 psql: could not connect to server: Connection refused
2016-11-22 14:36:09.227139  Is the server running on host "localhost" (::1) and accepting
2016-11-22 14:36:09.227156  TCP/IP connections on port 5432?
2016-11-22 14:36:09.227166 could not connect to server: Connection refused
2016-11-22 14:36:09.227187  Is the server running on host "localhost" (127.0.0.1) and accepting
2016-11-22 14:36:09.227196  TCP/IP connections on port 5432?
2016-11-22 14:36:09.261292 psql: could not connect to server: Connection refused
2016-11-22 14:36:09.261378  Is the server running on host "localhost" (::1) and accepting
2016-11-22 14:36:09.261396  TCP/IP connections on port 5432?
2016-11-22 14:36:09.261416 could not connect to server: Connection refused
2016-11-22 14:36:09.261426  Is the server running on host "localhost" (127.0.0.1) and accepting
2016-11-22 14:36:09.261439  TCP/IP connections on port 5432?

jovyan@6353d26f2fd8:/ConfinedWater$ psql
psql: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

Any hint?

As I don't don't particularly like Docker (on top of ubuntu@vmware@windows), I'd prefer to install and use Deepdive. I have three question:

(1) Is there an older version I can use which will work? (2) Any ETA on version 0.9? (3) Will v0.9 include a DD installer, or are you going to stick with Docker?

Thank you in advance! Keep up the good work.

netj commented 7 years ago

Sorry for my lagging answers:

  1. There's no older release that includes the deepdive-corenlp interface that can run the example in the master branch.
  2. The code is ready in master. Documentation needs a bit of update, which I'm trying to find time to work on.
  3. We will keep maintaining the installer with binary releases for 0.9 for folks who cannot use Docker, but Docker will be the recommended setup.

I see how Docker adds another layer of complexity to your setup. However, I believe it's the probably the best solution for both developers and users to create a uniform environment to build and depend on. Here's my recommendation: