Open profressor opened 7 years ago
Seems like you're using the example code on the master branch but the 0.8-STABLE version for DeepDive. We'll release 0.9 soon that can run the example on master. Meanwhile, I recommend running our Docker images: https://github.com/HazyResearch/deepdive/blob/master/doc/installation.md#launch-without-installing
Tried the docker images. Didnt work
user@vm:~$ bash <(curl -fsSL git.io/getdeepdive) ++ get-url https://github.com/HazyResearch/deepdive/raw/master/util/install.sh ++ curl -fsSL https://github.com/HazyResearch/deepdive/raw/master/util/install.sh
DeepDive installer for Ubuntu
1) deepdive 6) postgres 2) deepdive_docker_sandbox 7) postgres_xl 3) deepdive_example_notebook 8) run_deepdive_tests 4) deepdive_from_release 9) spouse_example 5) jupyter_notebook
Install what (enter to repeat options, a to see all, q to quit, or a number)? 2
Starting installation for deepdive_docker_sandbox
- get-url https://github.com/HazyResearch/deepdive/archive/v0.8-STABLE.tar.gz
- curl -fsSL https://github.com/HazyResearch/deepdive/archive/v0.8-STABLE.tar.gz
- tar xvzf - -C deepdive-0.8-STABLE.download deepdive-0.8-STABLE/sandbox deepdive-0.8-STABLE/test/postgresql/Dockerfile.postgres tar: deepdive-0.8-STABLE/sandbox: Not found in archive tar: deepdive-0.8-STABLE/test/postgresql/Dockerfile.postgres: Not found in archive tar: Exiting with failure status due to previous errors
Failed installation for deepdive_docker_sandbox
Install what (enter to repeat options, a to see all, q to quit, or a number)?
Is there a way to switch from stable to master build ? Or any other fix for this problem?
When will v0.9 be released?
@profressor Sorry but there was a glitch in the installer. It is now fixed, but for extra certainty, you can use the following command:
INSTALLER_BRANCH=master RELEASE=master bash <(curl -fsSL git.io/getdeepdive) deepdive_docker_sandbox
Using that command gives me another error
Step 5 : USER jovyan ---> Using cache ---> d04688cef943 Step 6 : ADD install.sh /deepdive/ ERROR: Service 'deepdive-notebooks' failed to build: lstat install.sh: no such file or directory
Failed installation for deepdive_docker_sandbox
Any ideas?
@profressor Thanks for spotting another glitch. The fix is just pushed now, and I made sure it works for me on a fresh Docker for Mac installation. This Docker sandbox we're introducing in this release, honestly, hasn't been tested thoroughly or used widely yet, so your early feedback is very helpful.
The Docker sandbox installation works now. However Im not familiar how to use it to build and run my own DD apps what I intend to do. Using my prefered method No. 1. (Deepdive installer) however shows some error
✗ partition_integers works (in test file util/test/partition_integers.bats, line 43) `partition_integers $m $n | check_correct_output $m $n' failed partitioning 0 by 2 /tmp/bats.72064.src: line 6: incorrect_lines: Permission denied ✓ partition_integers fails on non-positive partitions
125 tests, 1 failure, 5 skipped
Failed installation for run_deepdive_tests
Looks like a tiny perm. error which can be fixed easily. Back to topic, Im still stuck on the spouse example doing the sentences. Im using the updated version which still gives me (another) error:
user@vm:~/spouse$ deepdive compile (OK)
user@vm:~/spouse$ deepdive do sentences ‘run/RUNNING’ -> ‘20161118/002015.669047933’ ‘run/LATEST’ -> ‘20161118/002015.669047933’ 2016-11-18 00:20:19.019593 # on vm: deepdive do sentences 2016-11-18 00:20:19.019657 # run/20161118/002015.669047933/plan.sh 2016-11-18 00:20:19.019670 # execution plan for data/sentences 2016-11-18 00:20:19.019677 2016-11-18 00:20:19.019686 : ## process/init/app ########################################################## 2016-11-18 00:20:19.019693 : # Done: 2016-11-18T00:06:32+0100 (13m 43s ago) 2016-11-18 00:20:19.019699 : process/init/app/run.sh 2016-11-18 00:20:19.019705 : mark_done process/init/app 2016-11-18 00:20:19.019712 : ############################################################################## 2016-11-18 00:20:19.019721 2016-11-18 00:20:19.019727 2016-11-18 00:20:19.019733 : ## process/init/relation/articles ############################################ 2016-11-18 00:20:19.019739 : # Done: 2016-11-18T00:06:32+0100 (13m 43s ago) 2016-11-18 00:20:19.019745 : process/init/relation/articles/run.sh 2016-11-18 00:20:19.019750 : mark_done process/init/relation/articles 2016-11-18 00:20:19.019756 : ############################################################################## 2016-11-18 00:20:19.019764 2016-11-18 00:20:19.019770 2016-11-18 00:20:19.019776 : ## data/articles ############################################################# 2016-11-18 00:20:19.019782 : # Done: 2016-11-18T00:06:32+0100 (13m 43s ago) 2016-11-18 00:20:19.019787 : # no-op 2016-11-18 00:20:19.019793 : mark_done data/articles 2016-11-18 00:20:19.019800 : ############################################################################## 2016-11-18 00:20:19.019807 2016-11-18 00:20:19.019812 2016-11-18 00:20:19.019818 ## process/ext_sentences_by_nlp_markup ####################################### 2016-11-18 00:20:19.019825 # Done: N/A 2016-11-18 00:20:19.019831 process/ext_sentences_by_nlp_markup/run.sh 2016-11-18 00:20:19.019837 ++ dirname process/ext_sentences_by_nlp_markup/run.sh 2016-11-18 00:20:19.019844 + cd process/ext_sentences_by_nlp_markup 2016-11-18 00:20:19.019850 + export DEEPDIVE_CURRENT_PROCESS_NAME=process/ext_sentences_by_nlp_markup 2016-11-18 00:20:19.019856 + DEEPDIVE_CURRENT_PROCESS_NAME=process/ext_sentences_by_nlp_markup 2016-11-18 00:20:19.019862 + export DEEPDIVE_LOAD_FORMAT=tsv 2016-11-18 00:20:19.019869 + DEEPDIVE_LOAD_FORMAT=tsv 2016-11-18 00:20:19.019875 + deepdive compute execute 'input_sql= SELECT R0.id AS "articles.R0.id", R0.content AS "articles.R0.content" 2016-11-18 00:20:19.019883 FROM articles R0 2016-11-18 00:20:19.019889
2016-11-18 00:20:19.019894 ' 'command="$DEEPDIVE_APP"/udf/nlp_markup.sh' output_relation=sentences 2016-11-18 00:20:19.027645 Executing with the following configuration: 2016-11-18 00:20:19.027708 DEEPDIVE_NUM_PROCESSES=3 2016-11-18 00:20:19.027720 DEEPDIVE_NUM_PARALLEL_UNLOADS=1 2016-11-18 00:20:19.027727 DEEPDIVE_NUM_PARALLEL_LOADS=1 2016-11-18 00:20:19.027734 output_relation_tmp=dd_tmp_sentences 2016-11-18 00:20:19.027739 2016-11-18 00:20:19.116391 CREATE TABLE 2016-11-18 00:20:19.237686 CREATE TABLE 2016-11-18 00:20:19.275085 unloading to feed_processes-1: ' SELECT R0.id AS "articles.R0.id", R0.content AS "articles.R0.content" 2016-11-18 00:20:19.275149 FROM articles R0 2016-11-18 00:20:19.275160
2016-11-18 00:20:19.275168 ' > 2016-11-18 00:20:19.278372 No Bazaar/Parser set up at: /home/user/spouse/udf/bazaar/parser 2016-11-18 00:20:19.278409 /home/user/local/util/compute-driver/local/compute-execute: line 140: kill: (5382) - No such process 2016-11-18 00:20:19.279831 [ERROR] command='"$DEEPDIVE_APP"/udf/nlp_markup.sh': PID 5382: finished with non-zero exit status (0) 2016-11-18 00:20:19.279923 /home/user/local/util/compute-driver/local/compute-execute: line 138: 5383 Terminated DEEPDIVE_CURRENT_PROCESS_INDEX=$i bash -c "$command" < process-$i.input > process-$i.output 2016-11-18 00:20:19.279942 /home/user/local/util/compute-driver/local/compute-execute: line 138: 5384 Terminated DEEPDIVE_CURRENT_PROCESSINDEX=$i bash -c "$command" < process-$i.input > process-$i.output 2016-11-18 00:20:19.279949 /home/user/local/util/compute-driver/local/compute-execute: line 138: 5388 Terminated mkmimo process-.output > outputcomputed- 2016-11-18 00:20:19.279971 /home/user/local/util/compute-driver/local/compute-execute: line 138: 5389 Terminated deepdive-load "$output_relation_tmp" output_computed- 2016-11-18 00:20:19.279984 /home/user/local/util/compute-driver/local/compute-execute: line 138: 5395 Terminated deepdive-db unload "$input_sql" "$DEEPDIVE_LOAD_FORMAT" feed_processes- unloading: 0:00:00 0 [ 0 /s]([ 0 /s])il/compute-driver/local/compute-execute: line 138: 5396 Terminated mkmimo feed_processes-* > process-*.input unloading: 0:00:00 12KiB [ 546KiB/s]([ 546KiB/s]) ‘run/ABORTED’ -> ‘20161118/002015.669047933’
Can this be fixed? I need to run the spouse example in order to understand how DD works. I need to get it working to finish my own research on my masters thesis about DeepDive. Any help appreciated!
You should run the spouse example within the sandbox. The output you posted seems to be coming from the deepdive 0.8.x installed outside the sandbox. The sandbox includes the spouse example code and data with new version of deepdive that no longer relies on bazaar/parser. It should be very handy to try and apply to your own data.
Hi Jaeho,
Im using the sandbox image now but there is a problem connecting to the database. Im not too familiar with docker :/
bash <(curl -fsSL git.io/getdeepdive)
...
deepdive-notebooks_1 | [W 14:10:01.971 NotebookApp] Widgets are unavailable. Please install widgetsnbextension or ipywidgets 4.0
deepdive-notebooks_1 | [I 14:10:01.990 NotebookApp] Serving notebooks from local directory: /ConfinedWater
deepdive-notebooks_1 | [I 14:10:01.990 NotebookApp] 0 active kernels
deepdive-notebooks_1 | [I 14:10:01.990 NotebookApp] The Jupyter Notebook is running at: http://0.0.0.0:8888/
deepdive-notebooks_1 | [I 14:10:01.990 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
user@vm:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dc7ee2c58466 hazyresearch/postgres "/docker-entrypoint.s" 4 days ago Up 24 minutes 5432/tcp sandbox_database_1
user@vm:~$ docker run -i -t -v $HOME:/vm hazyresearch/deepdive:latest /bin/bash
jovyan@6353d26f2fd8:/vm/spouse$ ls
app.ddlog db.url deepdive.conf input run udf
jovyan@6353d26f2fd8:/vm/spouse$ deepdive compile
‘run/compiled’ -> ‘20161122/143702.284636268’
jovyan@6353d26f2fd8:/vm/spouse$ deepdive do sentences
‘run/RUNNING’ -> ‘20161122/143608.888898717’
2016-11-22 14:36:09.071365 process/ext_sentences_by_nlp_markup/run.sh
2016-11-22 14:36:09.071107 ################################################################################
2016-11-22 14:36:09.071216 # Host: 6353d26f2fd8
2016-11-22 14:36:09.071236 # DeepDive: deepdive v0.8.0-742-g4b812a1 (Linux x86_64)
2016-11-22 14:36:09.071247 export PATH='/usr/local/bin':"$PATH"
2016-11-22 14:36:09.071256 export DEEPDIVE_PWD='/vm/spouse'
2016-11-22 14:36:09.071265 export DEEPDIVE_APP='/vm/spouse'
2016-11-22 14:36:09.071278 cd "$DEEPDIVE_APP"/run
2016-11-22 14:36:09.071288 export DEEPDIVE_RUN_ID='20161122/143608.888898717'
2016-11-22 14:36:09.071297 # Plan: 20161122/143608.888898717/plan.sh
2016-11-22 14:36:09.071306 # Targets: sentences
2016-11-22 14:36:09.071315 ################################################################################
2016-11-22 14:36:09.071324
2016-11-22 14:36:09.071334 # process/init/app/run.sh ####################################### last done: 2016-11-21T23:38:35+0000 (14h 57m 33s ago)
2016-11-22 14:36:09.071344 # process/init/relation/articles/run.sh ######################### last done: 2016-11-21T23:38:35+0000 (14h 57m 33s ago)
2016-11-22 14:36:09.071353 # deepdive mark 'done' data/articles ############################ last done: 2016-11-21T23:38:35+0000 (14h 57m 33s ago)
2016-11-22 14:36:09.071365 process/ext_sentences_by_nlp_markup/run.sh ########################## last done: N/A
2016-11-22 14:36:09.071375 ++ dirname process/ext_sentences_by_nlp_markup/run.sh
2016-11-22 14:36:09.071384 + cd process/ext_sentences_by_nlp_markup
2016-11-22 14:36:09.071407 + : dd_tmp_ dd_old_
2016-11-22 14:36:09.071436 + export DEEPDIVE_CURRENT_PROCESS_NAME=process/ext_sentences_by_nlp_markup
2016-11-22 14:36:09.071467 + DEEPDIVE_CURRENT_PROCESS_NAME=process/ext_sentences_by_nlp_markup
2016-11-22 14:36:09.071487 + export DEEPDIVE_LOAD_FORMAT=tsv
2016-11-22 14:36:09.071524 + DEEPDIVE_LOAD_FORMAT=tsv
2016-11-22 14:36:09.071553 + output_relation=sentences
2016-11-22 14:36:09.071568 + output_relation_tmp=dd_tmp_sentences
2016-11-22 14:36:09.071579 + output_relation_old=dd_old_sentences
2016-11-22 14:36:09.071591 + deepdive create table-if-not-exists sentences
2016-11-22 14:36:09.192254 psql: could not connect to server: Connection refused
2016-11-22 14:36:09.192330 Is the server running on host "localhost" (::1) and accepting
2016-11-22 14:36:09.192347 TCP/IP connections on port 5432?
2016-11-22 14:36:09.192367 could not connect to server: Connection refused
2016-11-22 14:36:09.192377 Is the server running on host "localhost" (127.0.0.1) and accepting
2016-11-22 14:36:09.192390 TCP/IP connections on port 5432?
2016-11-22 14:36:09.227064 psql: could not connect to server: Connection refused
2016-11-22 14:36:09.227139 Is the server running on host "localhost" (::1) and accepting
2016-11-22 14:36:09.227156 TCP/IP connections on port 5432?
2016-11-22 14:36:09.227166 could not connect to server: Connection refused
2016-11-22 14:36:09.227187 Is the server running on host "localhost" (127.0.0.1) and accepting
2016-11-22 14:36:09.227196 TCP/IP connections on port 5432?
2016-11-22 14:36:09.261292 psql: could not connect to server: Connection refused
2016-11-22 14:36:09.261378 Is the server running on host "localhost" (::1) and accepting
2016-11-22 14:36:09.261396 TCP/IP connections on port 5432?
2016-11-22 14:36:09.261416 could not connect to server: Connection refused
2016-11-22 14:36:09.261426 Is the server running on host "localhost" (127.0.0.1) and accepting
2016-11-22 14:36:09.261439 TCP/IP connections on port 5432?
jovyan@6353d26f2fd8:/ConfinedWater$ psql
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Any hint?
As I don't don't particularly like Docker (on top of ubuntu@vmware@windows), I'd prefer to install and use Deepdive. I have three question:
(1) Is there an older version I can use which will work? (2) Any ETA on version 0.9? (3) Will v0.9 include a DD installer, or are you going to stick with Docker?
Thank you in advance! Keep up the good work.
Sorry for my lagging answers:
deepdive-corenlp
interface that can run the example in the master branch. I see how Docker adds another layer of complexity to your setup. However, I believe it's the probably the best solution for both developers and users to create a uniform environment to build and depend on. Here's my recommendation:
docker-compose
so you can worry less on other things such as linking with database containers. If you want to directly operate at the Docker level, you must link a hazyresearch/postgres container as database
when running the hazyresearch/deepdive image.sandbox/workdir/
to /ConfinedWater/workdir/
in the container and keeps the database on host sandbox/database/
.
cd sandbox
docker-compose run --rm --entrypoint bash deepdive-notebooks
Hi, following your spouse tutorial, I managed to fix some issues on they way that popped up following exactly the instructions. However now Im stucked. Im at the spouse example (step "deepdive do sentences"). Using latest stable Deepdive, Postgresql, [l]Ubuntu 16.04.1
Any help appreciated. Also:
Running etc/deepdive_bash_completion.sh just does nothing. If that helps.
http://pastebin.com/uxQ7z7fM