augur run throws error: command not found

[Adarsh-NP]

Describe the bug: A clear and concise description of what the bug is. after installing augur in my system which is ubuntu 19.10, first of all I get the warning

after that when I try to run augur using the command augur run I get this error

Kindly help

[ccarterlandis]

Hi @Adarsh-NP, can you share your logs/backend.log file? That should have more detail about what went wrong. That Clock skew detected error also seems weird...I've never seen that before. Hopefully the log files will provide some more clues!

[Adarsh-NP]

@ccarterlandis here are the backend-install.err and backend-install.log files

backend-install.log

backend-install.txt

[ccarterlandis]

@Adarsh-NP based on the output of backend-install.log, it looks to me like you're on the master branch. That branch is pretty outdated; I suggest checking out the dev branch locally, as I've cleaned up a lot of the problems with the installation process (and many others) there.

After you've switched to dev, I recommend running make clean, and then running make install to make sure all the old build files get removed. If that doesn't work, you could try recreating your virtual environment. That ModuleNotFound: _ctypes error is interesting to me though. Are you using Ubuntu?

[Adarsh-NP]

Yes, I am using Ubuntu 19.10

On Fri, 13 Mar 2020, 04:44 Carter Landis, notifications@github.com wrote:

@Adarsh-NP https://github.com/Adarsh-NP based on the output of backend-install.log, it looks to me like you're on the master branch. That branch is pretty outdated; I suggest checking out the dev branch locally, as I've cleaned up a lot of the problems with the installation process (and many others) there.

After you've switched to dev, I recommend running make clean, and then running make install to make sure all the old build files get removed. If that doesn't work, you could try recreating your virtual environment. That ModuleNotFound: _ctypes error is interesting to me though. Are you using Ubuntu?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chaoss/augur/issues/601#issuecomment-598472658, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDTEONHYGOJ2GWTSTMMPLTRHFUDTANCNFSM4LF5SUQA .

[ccarterlandis]

Okay, we support Ubuntu 16.04 LTS and later (the latest I've personally tested is 18.04 LTS, but it stands to reason 19.10 should work as well). Have you been able to run the commands on the dev branch, and do you see the same output?

[Adarsh-NP]

Yes, On dev branch make commands are running but augur run still throws errors and not running.

On Mon, 16 Mar 2020, 21:36 Carter Landis, notifications@github.com wrote:

Okay, we support Ubuntu 16.04 LTS and later (the latest I've personally tested is 18.04 LTS, but it stands to reason 19.10 should work as well). Have you been able to run the commands on the dev branch, and do you see the same output?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chaoss/augur/issues/601#issuecomment-599620782, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDTEOLOGCLSWJ6FON53LMDRHZFAFANCNFSM4LF5SUQA .

[ccarterlandis]

Hmmm, okay. Are the logs any different, and if so could you share them again? 😁

[Adarsh-NP]

Yeah sure Just a minute

On Mon, 16 Mar 2020, 21:44 Carter Landis, notifications@github.com wrote:

Hmmm, okay. Are the logs any different, and if so could you share them again? 😁

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chaoss/augur/issues/601#issuecomment-599625169, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDTEOIKJDUTAXVDNOSHTYDRHZF6NANCNFSM4LF5SUQA .

[Adarsh-NP]

attaching backend-install.err, backend-install.log and backend.log

backend-install.log backend.log

backend-install.txt

Also this time there were more files than the ones after previous installation, is it due to reinstallation, but that should have overridden the previous installation right?

Thanks in advance.

[ccarterlandis]

Seems there was an error downloading the urllib3 package:

pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

To be honest, I've never seen this error before. My first guess is that it's getting timed out because it's taking too long to download for some reason; can you think of anything that could be causing a slowdown like a VPN, proxy, or just poor internet connection? You could try setting the timeout to be higher than the default to see if that fixes anything (I will leave the details of doing that to you 😉 )

[NodiraIbrogimova]

I'm getting error while I run augur run as well. With information written here, I tried to run frontend and backend separately. make frontend throws: make: Nothing to be done for frontend And the documentation link in make frontend section isn't working. I was able to start frontend with npm run start (or just prompting from package.lock) on localhost, but for backend, I can't find any way to run without error.

P.S. I read your comments above. The installation is not getting much time, maybe internet is working well. But, if it requires a really fast internet connection then it's probably impossible here.

Can you please help @ccarterlandis ?

[NodiraIbrogimova]

@Adarsh-NP were you able to fix the error? If yes, can you please share with your solution?

[Adarsh-NP]

No I still haven't been able to, I'll do it now, last few days went in travelling to hometown because university decided to send students home in the view of COVID-19.

But did you try switching to the Dev branch and run the command make clean and then make install. See if it fixes the issue. More detailed information is given in this thread above by Carter, refer to that in case of any issues. Also if you run into new issues kindly share in this thread so that we could help each other if need be.

Regards

On Tue, 17 Mar 2020, 09:41 Nodira Ibragimova, notifications@github.com wrote:

@Adarsh-NP https://github.com/Adarsh-NP were you able to fix the error? If yes, can you please share with your solution?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chaoss/augur/issues/601#issuecomment-599868117, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDTEOK2TGUXNBMHJMZEQTTRH3Z6FANCNFSM4LF5SUQA .

[Adarsh-NP]

Hello @NodiraIbrogimova , yeah, so I deleted the cloned repo, cloned it again and switched to dev branch immediately on the first run and ran make clean after that make install and it is working perfectly fine ie augur run now runs. Thanks @ccarterlandis for helping me throughout and I hope this helps you @NodiraIbrogimova.

[Adarsh-NP]

Also, @ccarterlandis should I close this issue, since it seems to be resolved as of now.

[NodiraIbrogimova]

Dear @alexcourouble, thanks a lot for providing information. Yes, I'm working on dev branch and tried install and reinstall loop several times :D Also, checked some other ways in documentation given here I'll try one more time with suggestions and will let you guys know here. Can you please keep the issue open because the issue isn't solved in my case yet.

[NodiraIbrogimova]

Repeating doing the installation process from the beginning. Recent two tries for:

1. deleting repo from development environment
2. checkout to dev branch
3. make clean
4. make install didn't work :/ I think maybe I'm choosing the wrong options and that might be throwing error. For example, one question is about the option in screenshot below. Which options should I choose? What does this do? Clone remote repository? I see facade_worker in workers directory in augur project. But, in installation message it's asking for repository (remote). I'm a bit confused. Help me someone please, blowing my brain with installation failure for several days already :D

For Use an existing directory option getting an error:

@ccarterlandis :) @Adarsh-NP maybe can you share with your experience with installation steps, please?

[Adarsh-NP]

Hello @NodiraIbrogimova, I will surely try to help you out. I did not mean deleting it from environment, I just went to the folder and deleted that out of frustation but that helped :P So first of all I assume that you started the installation in a virtual environment, if not then that's highly suggested because of obvious reasons as the extra dependencies make the core of operating system heavy. So clone the repo and go to the augur directory. create and activate your virtual environment in the new cloned repo switch to the dev branch now run the following make clean make install For the option of using the existing directory, you have to provide with the absolute path of your Facade_worker that is in workers directory or augu r. Try doing that, it worked for me and I hope it works for you as well. Also, did you follow the instructions and create the psql database with given name, user and password? Also, it'll be better if you could provide us with your logs/backend.log file, that'll be helpful.

[NodiraIbrogimova]

@Adarsh-NP

So first of all I assume that you started the installation in a virtual environment, if not then that's highly suggested because of obvious reasons as the extra dependencies make the core of operating system heavy.

Yes, I followed instructions in documentation.

Also, did you follow the instructions and create the psql database with a given name, user and password?

I installed Postgres.app and started running the server. Now I have the following view in Postgres app:

I don't have prior experience with practicing database, no experience in backend. I'm having problem with Data collection and Backend part of Installation

For example, how should I complete this:

• database name (AUGUR_DB_NAME)
• host (AUGUR_DB_HOST)
• user (AUGUR_DB_USER)
• password (AUGUR_DB_PASSWORD)
• port (AUGUR_DB_PORT)

Some steps have little difference for my case as I use macOS, not Ubuntu. I'm a bit confused. logs/backend.log file for make install?

[NodiraIbrogimova]

Here are logs outputs: backend-installation.log frontend-installation.log workers-installation.log

@Adarsh-NP @ccarterlandis

[NodiraIbrogimova]

@Adarsh-NP @ccarterlandis @germonprez

Here I provided detailed information about my steps in augur directory (in virtual environment) hoping to get help and give you guys more information to discover what's wrong in my installation process. In addition, the log files for this installation are provided in prev. comment.

Process step-by-step:

# to create the environment
python3 -m venv $HOME/.virtualenvs/augur_env

# to activate it in bash
source $HOME/.virtualenvs/augur_env/bin/activate

and

1. make clean 2. make install 3. Choosing option 2) for

1) Would you like create the Augur database, user and schema LOCALLY?
2) Would you like to add the Augur schema to an already created Postgres 10 or 11 database (local or remote)?
3) Would you like to connect to a database already configured with Augur's schema? 

4.

Database: augur
Host: 
Port: 5433
User: augur
Password: [password]

5. Choosing option 2) for

The Facade data collection worker needs to clone repositories to run its analysis.
Would you like to use an existing directory, or create a new one?
1) Create a new directory
2) Use an existing directory

6. Giving absolute path for continuing step in 5

/Users/<..........>/augur/workers/facade_worker

7. Result after step 6

8. Choosing option 1) for:

Would you like to load your database with some sample data provided by Augur?
1) Yes
2) No

9. Result after step 8 :

10. Prompted augur after all of this (even though got some errors in prev. steps :D ) and got the errors:

[Adarsh-NP]

Port is 5432, I don't know if 5433 works Also, what about the host? Did you leave it blank?

On Wed, 18 Mar 2020, 11:49 Nodira Ibragimova, notifications@github.com wrote:

Here I provided detailed information about my steps in augur directory (in virtual environment) hoping to get help and give you guys more information to discover what's wrong in my installation process:

1. make clean 2. make install 3. Choosing option 2) for

1) Would you like create the Augur database, user and schema LOCALLY? 2) Would you like to add the Augur schema to an already created Postgres 10 or 11 database (local or remote)? 3) Would you like to connect to a database already configured with Augur's schema?

4.

Database: augur Host: Port: 5433 User: augur Password: [password]

5. Choosing option 2) for

The Facade data collection worker needs to clone repositories to run its analysis. Would you like to use an existing directory, or create a new one? 1) Create a new directory 2) Use an existing directory

6. Giving absolute path for continuing step in 5

/Users/<..........>/augur/workers/facade_worker

7. Result after step 6

[image: Screen Shot 2020-03-18 at 11 09 14 AM] https://user-images.githubusercontent.com/11291840/76930725-f9cf7900-6908-11ea-8b70-d1b4c3942556.png

8. Choosing option 1) for:

Would you like to load your database with some sample data provided by Augur? 1) Yes 2) No

9. Result after step 8 : [image: Screen Shot 2020-03-18 at 11 12 35 AM] https://user-images.githubusercontent.com/11291840/76930913-65b1e180-6909-11ea-8fdb-52ee5c5310df.png

10. Prompted augur after all of this (even though got some errors in prev. steps :D ) and got the errors:

[image: Screen Shot 2020-03-18 at 11 13 41 AM] https://user-images.githubusercontent.com/11291840/76931125-d3f6a400-6909-11ea-9047-165ca6ff65fe.png

[image: Screen Shot 2020-03-18 at 11 13 54 AM] https://user-images.githubusercontent.com/11291840/76931122-d35e0d80-6909-11ea-8676-16777c8543ce.png

[image: Screen Shot 2020-03-18 at 11 14 05 AM] https://user-images.githubusercontent.com/11291840/76931111-cf31f000-6909-11ea-91d4-f98364cc79f4.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chaoss/augur/issues/601#issuecomment-600445053, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDTEOMYHLVGFHZ5U7NUSD3RIBRVVANCNFSM4LF5SUQA .

[NodiraIbrogimova]

Yes. I left it blank.

Database: augur
Host: 
Port: 5433
User: augur
Password: [password]

[NodiraIbrogimova]

Couldn't connect to port 5432 from Postgres app, it's throwing popup message saying the port is busy. That's why tried port 5433 in installation process to prevent similar error 🙈

[NodiraIbrogimova]

After make clean, tried with port 5432 now. It's throwing the same errors above (for all steps).

[ccarterlandis]

Hi @NodiraIbrogimova, sorry I have not been as responsive these past few days. Let me try to clarify a few things.

1. You are correct in choosing option 2 (DB without the schema). With regards to credentials, since your database is running locally, your host should be localhost. Similarly, unless you have changed PostgreSQL's default port, your port should be 5432; you can check your configuration by under the "Server settings" dialog box when using Postgres.app. When you enter the correct credentials, the script should create the augur user you, so that error should go away.
2. We recommend using a dedicated folder that lives outside the augur/ repository for your Facade data collection. This is so that the files do not get accidentally checked into Git, and if you need to reset all your data collection progress, you'll need to remove all the clones made by Facade; this is easier to do in its own folder IMO, since you don't have to worry about accidentally deleting the code for the facade_worker.
3. I'm not sure if loading the sample data still works, as we have actually switched to using Docker to distribute a sample dataset that we also use for testing purposes.

I would also like to extend my sincere apologies for the difficult you've been experiencing. Myself and the rest of the Augur team have been working very diligently to make a great many enhancements to our system. However, I have not been doing things properly and updating the documentation as I go, and have instead let it fall very far out of date for far too long. As a fellow programmer I wholeheartedly understand the aggravation that comes with missing, incorrect, or otherwise badly written documentation. I do hope you can forgive me for this.

However, @CMPerniciaro and I have been hard at work updating our documentation to reflect the many enhancements and stability features we've made, and we hope to have these ready to merge into dev very soon. I will be sure to request your review on it, as the more eyes on it the better!

@Adarsh-NP I would also like to thank you sincerely for your assistance while I was unable to help! 🎉

[ccarterlandis]

I closed this on accident while trying to type out my above comment. I will keep it open until we are able to get you up and running as well @NodiraIbrogimova!

[NodiraIbrogimova]

@ccarterlandis thank you so much for a detailed answer! :D 🎉At night I was able to fix the problem with the server. But, I got an error when I run augur run. After your comment I think it might be because of the installation with sample data from augur.

I'm not sure if loading the sample data still works, as we have actually switched to using Docker to distribute a sample dataset that we also use for testing purposes.

One question here is, does the sentence above mean I've better choose No for the option below?

Would you like to load your database with some sample data provided by Augur?
1) Yes
2) No

And if I got you right

We recommend using a dedicated folder that lives outside the augur/ repository for your Facade data collection.

means to use absolute path /Users/<some folders here>/augur/workers/facade_worker for facade_worker which is located in root augur folder? :D

[NodiraIbrogimova]

Yaaaaaay! It worked! 🎉🎉🎉 Now I can start the frontend and backend servers together. Thank you, guys! @ccarterlandis @Adarsh-NP

[ccarterlandis]

One question here is, does the sentence above mean I've better choose No for the option below?

That is correct.

If you're familiar with Docker, and want to try the test database out, you can use the following command to run the container:

docker run -d -p 5434:5432 --name augur_test_database augurlabs/augur:test_data

You'll then need to change the Database section of your augur.config.json to:

    "Database": {
        "name": "test_data",
        "host": "localhost",
        "key": "your_github_access_token_here",
        "password": "augur",
        "port": 5434,
        "schema": "augur_data",
        "user": "augur"
    }

You could change the port from 5434 if you want, I just use it because it's close to the default PostgreSQL port.

..means to use absolute path /Users//augur/workers/facade_worker

I would recommend explicitly not using this path. For reference, I use /Users/carter/projects/work/repos/ as my repo directory, and my facade_worker installation lives at /Users/carter/projects/work/augur/facade_worker/. That way when the facade_worker clones the Git repositories to your machine, it doesn't clone into Augur's source code. If it did that things would get pretty messy pretty fast, since you'd be putting a Git repo inside a Git repo.

I'm glad you were able to get it working! 🎉 @NodiraIbrogimova

[Adarsh-NP]

Ok, so I loaded the data. What should I do now because in the documentation it was mentioned that we should load the data. 😅😅

On Thu, 19 Mar 2020, 11:03 Carter Landis, notifications@github.com wrote:

One question here is, does the sentence above mean I've better choose No for the option below? Yes, the correct option would be no.

If you're familiar with Docker, and want to try the test database out, you can use the following command to run the container:

docker run -d -p 5434:5432 --name augur_test_database augurlabs/augur:test_data

You'll then need to change the Database section of your augur.config.json to:

"Database": {

    "connection_string": "sqlite:///:memory:",

    "name": "test_data",

    "host": "localhost",

    "key": "your_github_access_token_here",

    "password": "augur",

    "port": 5434,

    "schema": "augur_data",

    "user": "augur"

}

You could change the port from 5434 if you want, I just use it because it's close to the default PostgreSQL port.

..means to use absolute path /Users//augur/workers/facade_worker

I would recommend explicitly not using this path. For reference, I use /Users/carter/projects/work/repos/ as my repo directory, and my augur installation lives at /Users/carter/projects/work/augur/facade_worker/. That way when the facade_worker clones the Git repositories to your machine, it doesn't clone into Augur's source code. If it did that things would get pretty messy pretty fast, since you'd be putting a Git repo inside a Git repo.

I'm glad you were able to get it working! 🎉

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chaoss/augur/issues/601#issuecomment-600996830, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDTEOP3AZRXZ4GS6YLEMETRIGVCBANCNFSM4LF5SUQA .

[NodiraIbrogimova]

Before you close the issue, I would like to share the steps I took during installation here in case someone else faces the same difficulty like I had. As my issue is related to this issue, @ccarterlandis can I post it here as a comment, please?

[Adarsh-NP]

I feel that's a good thing to do considering we ran into the same issue. Good thinking, but it would be better if we document it and attach it in the documentation in a separate link maybe called as issues while installation. Will help a lot of people. What say Carter?

On Thu, 19 Mar 2020, 11:24 Nodira Ibragimova, notifications@github.com wrote:

Before you close the issue, I would like to share the steps I took during installation here in case someone else faces the same difficulty like I had. As my issue is related to this issue, @ccarterlandis https://github.com/ccarterlandis can I post it here as a comment, please?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chaoss/augur/issues/601#issuecomment-601001857, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANDTEOLIIJEDVRDTTAM4AJLRIGXQVANCNFSM4LF5SUQA .

[ccarterlandis]

@Adarsh-NP That is no problem. It's just a different dataset :) If you are doing development you will need to use the Docker image for testing, and if you want I can send you a link to first version of the documentation on how to set that up after I publish it.

@NodiraIbrogimova (and @Adarsh-NP) please feel free to share any feedback, notes or steps you have from your experience installing Augur. It'll really help me make our documentation a lot better to know where the specific pain points are from people who aren't used to all the pain points like I am 😂

I will actually go ahead and open draft PR from our documentation branch into dev. That way you can compare your feedback to what I've already got written of the new instructions (spoiler: it's not a whole lot at the moment), and we can capture all the discussion around the new docs in a central location. 😁

[NodiraIbrogimova]

@ccarterlandis said they are doing some changes in documentation. I'm afraid there might be some mess if they are making changes in the installation part as well.

If the changes we do don't conflict with the ones you guys (@ccarterlandis and @CMPerniciaro) are doing, I would be happy to contribute in this so people could directly see the changes from there. Otherwise, for the time being I can post it here in the comment and you could review and make some changes (maybe before we add them to documentation).

P.S. I thought it was a bad internet connection at the beginning, but it seems some links in the documentation are not working.

[NodiraIbrogimova]

@ccarterlandis do we have some group chat in slack or another platform? It would be great to have one for discussion (in case we don't have). I checked one link for slack somewhere in the description, but I couldn't open it.

[Adarsh-NP]

@NodiraIbrogimova, yeah some links in the documentation are broken actually and our mentors are up on that. For the time being lets create a document together on the installation issues we faced using google docs, what say? Or commenting here would also help.

[Adarsh-NP]

@Adarsh-NP That is no problem. It's just a different dataset :) If you are doing development you will need to use the Docker image for testing, and if you want I can send you a link to first version of the documentation on how to set that up after I publish it. Yeah if possible, that would be helpful, although I have no experience with docker yet. But for the time being, what to do as of now, like since I've loaded the data and you said about the possibility of a .git file in .git (I've run into those errors while working for some of my personal projects XD) so what about writing the code for interaction with gitlab api.

[NodiraIbrogimova]

Great, how about moving to google docs for working on documentation? We could see changes by each other and discuss commenting. It's better to move this discussion to a different place, IMO. That's just one option, feel free to let me know about your point of view. By the way, can you share your e-mail, please? Mine is: nodira.ibrogimova@gmail.com

[Adarsh-NP]

Yeah, even I feel so hence I mentioned it. My mail is: f20180806@goa.bits-pilani.ac.in

[ccarterlandis]

@Adarsh-NP @NodiraIbrogimova I've gone ahead an opened a PR with what I've currently got written, but I think starting with Google Docs is a really great idea. It would be easier to consolidate your notes there, since it's all in real-time. If you like, I can also share a link to the document you create in the PR, and we can all work together there. Up to you 😁

[Adarsh-NP]

Sure, we'll share the link.

[NodiraIbrogimova]

@ccarterlandis yes, that's even better :D We can have your eyes on it while making some changes

[ccarterlandis]

I will go ahead and close this issue, but feel to use this thread or #616 to continue the discussion. I am excited to see what you all come up with! 🚀

[ccarterlandis]

I will also work on getting everything set up in our Slack channel 👍

[NodiraIbrogimova]

@ccarterlandis can you share the channel link please?

[ccarterlandis]

@NodiraIbrogimova I've got to double check our permissions and then set up our guest access correctly, which I'll do in the morning (it's 1:30am where I am). I will post it in the main GSoC thread, and ping both you and @Adarsh-NP when I do so :)

[ArcD7]

Hey @Adarsh-NP @ccarterlandis @NodiraIbrogimova . I hope you are doing well. I followed your instructions for installation but when i try make install on the dev branch i get this error But, when I try it without changing the branch I get no error.

[NodiraIbrogimova]

Hello @ArcD7! Can you share your backend.logs file, please? You can find the file in logs/backend.logs of the augur root directory.

[ArcD7]

Here you go! backend-installation.log