search

SiRumCz / CSC501

CSC501 assignments
0 stars 1 forks source link

get neo4j running #64

Closed jonhealy1 closed 4 years ago

SiRumCz commented 4 years ago

looks nice, great work!

jonhealy1 commented 4 years ago

Hi thanks! Hopefully it works for everyone. Should do.

SiRumCz commented 4 years ago

For some reasons, I am having problems with the data. My output from python load_data.py is below:

<py2neo.database.Cursor object at 0x7fd613d9d978>

     ID Source Target  Time
0  None   None   None  None
1  None   None   None  None
2  None   None   None  None

  Source  numCount
0   None         0

Do you know if there is anything I did wrong?

Kevin

jonhealy1 commented 4 years ago

I'm having problems with hosting the .csv files on filebin. It seems like I have to reload the data up to filebin to get it to work.

When I try to run the .csv files locally I get this: py2neo.database.ClientError: ExternalResourceFailed: Couldn't load the external resource at: file:/var/lib/neo4j/import/body_tab_separated.csv

I haven't figured it out yet ...

jonhealy1 commented 4 years ago

I made .csv from the links .tsv. I added the tab-separated fields and a couple of the comma separated ones for fun. There is positive, negative and composite sentiment according to VADER. Not sure what that means.

SiRumCz commented 4 years ago

I'm having problems with hosting the .csv files on filebin. It seems like I have to reload the data up to filebin to get it to work.

When I try to run the .csv files locally I get this: py2neo.database.ClientError: ExternalResourceFailed: Couldn't load the external resource at: file:/var/lib/neo4j/import/body_tab_separated.csv

I haven't figured it out yet ...

I have fixed this problem by adding -v $PWD:/var/lib/neo4j/import to the docker run command. 

sudo docker run --publish=7474:7474 --publish=7687:7687 --volume=$HOME/neo4j/data:/data -v $PWD:/var/lib/neo4j/import neo4j
SiRumCz commented 4 years ago

However, I am still getting some None values in the results. here is from load_data_body.py:

<py2neo.database.Cursor object at 0x7fd68f0c29b0>

     ID Source Target  Time
0  None   None   None  None
1  None   None   None  None
2  None   None   None  None

                      Source  numCount
0             subredditdrama      9330
1                circlebroke      4716
2            shitliberalssay      3936
3               outoftheloop      3916
4                  copypasta      3648
5             writingprompts      3414
SiRumCz commented 4 years ago

Here is the result from load_data_links.py

xxx$ python load_data_links.py 
<py2neo.database.Cursor object at 0x7f5fca2b2ef0>

     ID Source Target  Time a.Link_Sentiment
0  None   None   None  None             None
1  None   None   None  None             None
2  None   None   None  None             None

  Source  numCount
0   None         0
jonhealy1 commented 4 years ago

Nice job with the Docker command. I think for the load_data_body.py you're getting an empty result because a table has already been created with the same name so it has a bunch of empty rows at the head. You probably have to change the name of the table or erase the graphdb file and try again. The data is in there or you wouldn't get the count result. For load_data_links.py the code is still trying to load data from filebin. It will have to be changed to load the local title_tab_separated.csv file.

jonhealy1 commented 4 years ago

I think I fixed load_data_links.py but I don't have time to test it right now.

SiRumCz commented 4 years ago

Nice job with the Docker command. I think for the load_data_body.py you're getting an empty result because a table has already been created with the same name so it has a bunch of empty rows at the head. You probably have to change the name of the table or erase the graphdb file and try again. The data is in there or you wouldn't get the count result. For load_data_links.py the code is still trying to load data from filebin. It will have to be changed to load the local title_tab_separated.csv file.

I will try it later as I am away from my computer right now. thanks for the suggestion!

jonhealy1 commented 4 years ago

-v $PWD:/var/lib/neo4j/import

This isn't working for me it just makes Docker freeze I have no idea why.

SiRumCz commented 4 years ago

-v $PWD:/var/lib/neo4j/import

This isn't working for me it just makes Docker freeze I have no idea why.

for macos, it could be different command, but I am not sure. Here is where I found the answer: https://stackoverflow.com/questions/43606968/using-load-csv-to-import-a-local-file-to-neo4j-in-a-docker-container

SiRumCz commented 4 years ago

Maybe try this.

docker run \
--publish=7474:7474 --publish=7687:7687 \
--volume=$HOME/neo4j/data:/data \
--volume=$PWD:/var/lib/neo4j/import \
neo4j
jonhealy1 commented 4 years ago

docker run --publish=7474:7474 --publish=7687:7687 --volume=$HOME/neo4j/data:/data --volume=$PWD/import:/var/lib/neo4j/import neo4j

This worked on mac. Everything worked. I had to run docker from assignment3 folder and then I put .csv files in an import folder

jonhealy1 commented 4 years ago

I think the problem was that I wasn't running docker from the assignment 3 folder.

SiRumCz commented 4 years ago

docker run --publish=7474:7474 --publish=7687:7687 --volume=$HOME/neo4j/data:/data --volume=$PWD/import:/var/lib/neo4j/import neo4j

This worked on mac. Everything worked. I had to run docker from assignment3 folder and then I put .csv files in an import folder

I found out you put the raw data file in /imports folder, which leaves /import folder unuse, would'n that make the docker command to be this:

docker run \
--publish=7474:7474 --publish=7687:7687 \
--volume=$HOME/neo4j/data:/data \
--volume=$PWD/imports:/var/lib/neo4j/import \
neo4j

This worked on my ubuntu machine, but it failed on my mac machine :(

jonhealy1 commented 4 years ago

after wiped out my local graph databases, I successfully imported both body and title data

The folder should be called import not imports. Not sure why it worked for me...

soroushysfi commented 4 years ago

I am following the steps but when I try to put in the password(which is password) I get authentication failure. user name should be neo4j and connect url should be bolt://localhost:7687, right?

SiRumCz commented 4 years ago

default password is neo4j

On Sun., Oct. 27, 2019, 1:02 p.m. Soroush notifications@github.com wrote:

I am following the steps but when I try to put in the password(which is password) I get authentication failure. user name should be neo4j and connect url should be bolt://localhost:7687, right?

— You are receiving this because your review was requested. Reply to this email directly, view it on GitHub https://github.com/SiRumCz/CSC501/pull/64?email_source=notifications&email_token=AEWWYWTY5MTFKWECQ44ACO3QQXXTRA5CNFSM4JFLGM6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECLGUUQ#issuecomment-546728530, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEWWYWTBTVEJI5EDR4U7PJ3QQXXTRANCNFSM4JFLGM6A .

soroushysfi commented 4 years ago

I keep gettingImportError: No module named py2neo. I have installed the requirements and even tried to install py2neo module globally.

soroushysfi commented 4 years ago

@noonespecial009 Thanks for the good work it seems to be great. Unfortunately I don't know what's the problem with my system I can't run it.

jonhealy1 commented 4 years ago

What system are you on? Is the docker imager running? I just did pip install py2neo. Maybe try pip3 install py2neo?

jonhealy1 commented 4 years ago

conda install -c conda-forge py2neo (this might work too)

soroushysfi commented 4 years ago

What system are you on? Is the docker imager running? I just did pip install py2neo. Maybe try pip3 install py2neo?

I'm on Mac. Yeah the docker image is running but the problem is with the python modules. It's weird because I haven't had this problem before. I checked my Python libraries, I have py2neo and pandas installed but somehow in this project I get the error.

jonhealy1 commented 4 years ago

The only thing that I can think of is that you installed it for python 2 instead of 3? I'm using mac. I wonder what we should do now....

soroushysfi commented 4 years ago

I checked it's in python 3: Requirement already satisfied: py2neo in /Users/soroush/Library/Python/3.7/lib/python/site-packages (4.3.0) It's getting weird...

jonhealy1 commented 4 years ago

Do you know how to create a virtual environment? That might work

soroushysfi commented 4 years ago

Do you know how to create a virtual environment? That might work

I haven't tried it. I see if I can get it running with that, Thanks.

jonhealy1 commented 4 years ago

I could try to get everything running in docker compose

jonhealy1 commented 4 years ago

Have you guys used docker compose? I couldn't get the python container to talk to the neo4j one. I could it put it up as a branch. The neo4j container works fine.

soroushysfi commented 4 years ago

Have you guys used docker compose? I couldn't get the python container to talk to the neo4j one. I could it put it up as a branch. The neo4j container works fine.

I haven't worked with docker compose. I can read about it and see how it works