Source file not found

[victorvianna]

I was testing the code by trying to verify some simple pull requests. (I'm running through docker.) However when I open a pull request, I get the following error:

E902 IOError: [Errno 2] No such file or directory: '/src/foo.py'

[markstory]

Sounds like the worker's linting container doesn't have the application code mounted in. I don't personally run lintreview inside docker so I'm not entirely sure what is happening.

[victorvianna]

Even when I run without docker, it just hangs and doesn't post any comments to the pull request.

Some checks haven’t completed yet

[markstory]

Have you built the docker images that contain the required linting tools? Where does it hang if you enable debug logging?

[victorvianna]

[INFO] Listening at: http://127.0.0.1:5000 (23453) [INFO] Using worker: sync [INFO] Booting worker with pid: 23457

[victorvianna]

For docker, I've also tried to go back to the commit fcce6b8bb681e3bd6e35d440bf85258a32050fcd (the commit just before "Use image instead of building") and build the images myself, but he won't even find npm package with apt-get

[markstory]

So are you not able to build the images for lintreview? or the linter images (as they are separate).

[victorvianna]

Non-docker

Edit: this one's solved.

Docker

So are you not able to build the images for lintreview? or the linter images (as they are separate).

I am, with the current commit. I tried going back to a previous commit before the images were loaded from Docker Hub, to see if it would mount '/src' properly, but I may have checked out the wrong hash, I'm doing it again, I'll tell you soon.

[victorvianna]

Docker

Just tried commits 972024fe32c20dc1d3cae128cbdc28109c5d948d and ab8985ef9158ba0d687e97efa8605be4711d8f96 locally (manjaro 17.10) and I get the same thing for both, namely: 1) 'web' is classified as unhealthy when running "docker-compose ps" (no proper /ping response) 2) I see no src folder when running the code below:

docker-compose run web /bin/bash
ls /src

I've also tried 2 with 'worker' instead of 'web'.

[markstory]

@victorvianna Neither the web or consumer containers will have a /src directory. This directory only exists in the linter containers. The consumer spins up additional docker container for each linting tool that is run. Linters are run in separate containers as many of them execute code in the repository.

[victorvianna]

When replacing PEP8 by Flake8, it won't display the '/src/' problem anymore, except that it won't show any corrections.

Log output

worker_1  | [2018-08-13 15:09:39,857: INFO/Worker-1] Starting to process lint for victorvianna/webhooks/10
worker_1  | [2018-08-13 15:09:39,859: INFO/Worker-1] Loading pull request data from github. user=victorvianna repo=webhooks number=10
worker_1  | [2018-08-13 15:09:39,864: INFO/Worker-1] Building a url from ('https://api.github.com', 'repos', 'victorvianna', 'webhooks')
worker_1  | [2018-08-13 15:09:39,864: INFO/Worker-1] Missed the cache building the url
worker_1  | [2018-08-13 15:09:39,964: INFO/Worker-1] Attempting to get JSON information from a Response with status code 200 expecting 200
worker_1  | [2018-08-13 15:09:39,965: INFO/Worker-1] JSON was returned
worker_1  | [2018-08-13 15:09:39,969: INFO/Worker-1] Building a url from ('https://api.github.com/repos/victorvianna/webhooks', 'pulls', '10')
worker_1  | [2018-08-13 15:09:39,969: INFO/Worker-1] Missed the cache building the url
worker_1  | [2018-08-13 15:09:40,122: INFO/Worker-1] Attempting to get JSON information from a Response with status code 200 expecting 200
worker_1  | [2018-08-13 15:09:40,123: INFO/Worker-1] JSON was returned
worker_1  | [2018-08-13 15:09:40,125: INFO/Worker-1] Missed the cache building the url
worker_1  | [2018-08-13 15:09:40,246: INFO/Worker-1] Attempting to get JSON information from a Response with status code 201 expecting 201
worker_1  | [2018-08-13 15:09:40,246: INFO/Worker-1] JSON was returned
worker_1  | [2018-08-13 15:09:40,248: INFO/Worker-1] Cloning repository 'https://github.com/victorvianna/webhooks.git' into '/tmp/workspace/victorvianna/webhooks/10'
worker_1  | [2018-08-13 15:09:40,594: INFO/Worker-1] Checking out '478e862f696148ee26e0d17b26b91862178ddafe'
worker_1  | [2018-08-13 15:09:40,600: INFO/Worker-1] Loading pull request patches from github.
worker_1  | [2018-08-13 15:09:40,600: INFO/Worker-1] Building a url from ('https://api.github.com/repos/victorvianna/webhooks/pulls/10', 'files')
worker_1  | [2018-08-13 15:09:40,600: INFO/Worker-1] Missed the cache building the url
worker_1  | [2018-08-13 15:09:40,674: INFO/Worker-1] Attempting to get JSON information from a Response with status code 200 expecting 200
worker_1  | [2018-08-13 15:09:40,675: INFO/Worker-1] JSON was returned
worker_1  | [2018-08-13 15:09:40,676: INFO/Worker-1] Building a url from ('https://api.github.com/repos/victorvianna/webhooks/pulls/10', 'commits')
worker_1  | [2018-08-13 15:09:40,677: INFO/Worker-1] Missed the cache building the url
worker_1  | [2018-08-13 15:09:40,678: INFO/Worker-1] Running lint tools on 1 files
worker_1  | [2018-08-13 15:09:40,679: INFO/Worker-1] Running flake8 on 1 files
worker_1  | [2018-08-13 15:09:40,679: INFO/Worker-1] Running python2 container
worker_1  | [2018-08-13 15:09:41,766: INFO/Worker-1] Building a url from ('https://api.github.com/repos/victorvianna/webhooks/pulls/10', 'comments')
worker_1  | [2018-08-13 15:09:41,766: INFO/Worker-1] Missed the cache building the url
worker_1  | [2018-08-13 15:09:41,826: INFO/Worker-1] Attempting to get JSON information from a Response with status code 200 expecting 200
worker_1  | [2018-08-13 15:09:41,826: INFO/Worker-1] JSON was returned
worker_1  | [2018-08-13 15:09:41,826: INFO/Worker-1] Publishing review of 0 new comments for victorvianna/webhooks/pull/10
worker_1  | [2018-08-13 15:09:41,826: INFO/Worker-1] Building a url from ('https://api.github.com/repos/victorvianna/webhooks', 'statuses', '478e862f696148ee26e0d17b26b91862178ddafe')
worker_1  | [2018-08-13 15:09:41,956: INFO/Worker-1] Attempting to get JSON information from a Response with status code 201 expecting 201
worker_1  | [2018-08-13 15:09:41,957: INFO/Worker-1] JSON was returned
worker_1  | [2018-08-13 15:09:41,958: INFO/Worker-1] Completed lint processing for victorvianna/webhooks/10
worker_1  | [2018-08-13 15:09:41,963: INFO/Worker-1] Cleaned up pull request victorvianna/webhooks/10
worker_1  | [2018-08-13 15:09:41,966: INFO/MainProcess] Task lintreview.tasks.process_pull_request[7c147c1d-6959-49a4-9b12-904298d35aa3] succeeded in 2.10858612496s: None

[markstory]

worker_1 | [2018-08-13 15:09:41,826: INFO/Worker-1] Publishing review of 0 new comments for victorvianna/webhooks/pull/10

Is the key line here. Are you certain that your pull request contains lines that would trigger flake8 errors? lintreview only looks at lines that were changed in the pull request. If you set the logging level to DEBUG you'll get more output from the linter output and which errors survived filtering.

[victorvianna]

Are you certain that your pull request contains lines that would trigger flake8 errors?

Yes. I found the exact command run by the worker (via log.info) and ran it directly from my machine. In this case, the linter works, for example as in:

echo "  x== 2 " > /tmp/path_to_pull/foo.py
docker run --rm -v /tmp/path_to_pull/:/src python2 flake8 --isolated /src/foo.py
# With this code, the linter works

But it won't work if another container tries to run the image. The problem appears to be with the mounting of the folders and it is independent of the image you use (I tried replacing the python2-alpine by a similar Ubuntu 16.04 and got that same result). Ex:

user@pc:/home/user/lint-review $ docker-compose run -it worker /bin/bash
root@worker:/code # 
root@worker:/code # touch /tmp/bar.py
root@worker:/code # docker run -it -v /tmp:/src some_image /bin/bash
(/bin/bash will not be found for python:2.7-alpine, but you can use python or whatever)
root@some_image:/src#
root@some_image:/src# ls
(Won't show file bar.py)

[kostiq]

Hello @markstory. I have the same issue. Could you tell me status of this issue now?

[markstory]

@kostiq I don't know how to reproduce it so its stalled.

[kostiq]

@markstory Firstly, I was run docker-compose run web lintreview register user repo with -e LINTREVIEW_SERVER_NAME=my_server:port, otherwise flask + gunicorn was unhealthy.

Secondly, in docker-compose I changed worker for start with log in file and log_level = DEBUG

worker:
    image: markstory/lint-review
    command:
      - celery
      - -A
      - lintreview.tasks
      - worker
      - -f
      - log.txt
      - -l
      - debug
      - --concurrency
      - '1' 

Then PR was created, worker's log file output was:

[2018-10-11 07:09:17,329: INFO/Worker-1] Cloning repository 'https://github.com/kostiq/task.git' into '/tmp/workspace/kostiq/task/13'
[2018-10-11 07:09:17,329: DEBUG/Worker-1] Running ['git', 'clone', u'...github.com/kostiq/task.git', u'/tmp/workspace/kostiq/task/13']
[2018-10-11 07:09:18,307: INFO/Worker-1] Checking out '7d530b8316999b318549f313d9b3fb0cf2c72437'
[2018-10-11 07:09:18,308: DEBUG/Worker-1] Changing directories to /tmp/workspace/kostiq/task/13
[2018-10-11 07:09:18,308: DEBUG/Worker-1] Running ['git', 'checkout', u'7d530b8316999b318549f313d9b3fb0cf2c72437']
...
[2018-10-11 07:09:18,489: INFO/Worker-1] Running python3 container
[2018-10-11 07:09:18,490: DEBUG/Worker-1] Running ['docker', 'run', '--rm', '-v', u'/tmp/workspace/kostiq/task/13:/src', 'python3', 'pep8', '-r', '/src/union_lists.py']
[2018-10-11 07:09:19,258: DEBUG/Worker-1] Container output was: /usr/local/lib/python3.6/site-packages/pep8.py:2124: UserWarning: 

pep8 has been renamed to pycodestyle (GitHub issue #466)
Use of the pep8 tool will be removed in a future release.
Please install and use `pycodestyle` instead.

$ pip install pycodestyle
$ pycodestyle ...

  '\n\n'
/src/union_lists.py:1:1: E902 FileNotFoundError: [Errno 2] No such file or directory: '/src/union_lists.py'

Due to - /var/run/docker.sock:/var/run/docker.sock in docker-compose docker command from worker executes in host, but I don't understand where is files from git clone command. In host and worker folder /tmp/workspace/kostiq/task/... is empty.

My .lintrc:

[tools]
linters = flake8, pep8

[tool_pep8]
python = 3

[tool_flake8]
python = 3

[markstory]

Ok. You're likely going to have to debug this on your own as reviews worked locally for me last time I tried (earlier in this thread).

The consumer first clones your repository with

[2018-10-11 07:09:17,329: DEBUG/Worker-1] Running ['git', 'clone', u'...github.com/kostiq/task.git', u'/tmp/workspace/kostiq/task/13']

It then runs the following docker command:

[2018-10-11 07:09:18,490: DEBUG/Worker-1] Running ['docker', 'run', '--rm', '-v', u'/tmp/workspace/kostiq/task/13:/src', 'python3', 'pep8', '-r', '/src/union_lists.py']

Are you able to run those commands inside the consumer container to see if you can reproduce the missing/bad volume mount?

[kostiq]

Yeah. In consumer I run following commands:

git clone

check files

And docker run with volumes

[markstory]

Ok. So until you can fix that lintreview will not work correctly. It seems like the volume mount from the consumer into the tool container is not working correctly.

[kostiq]

Ok. I will try.