search

EpicWink / proxpi

PyPI caching proxy
MIT License
138 stars 25 forks source link

Statistics Report/dashboard #60

Open harshguptaserver opened 3 weeks ago

harshguptaserver commented 3 weeks ago

aim: using EpicWink/proxpi as a cache server for pip installs during container builds in the complete internal network

I would like to kindly request for a url/endpoint/feature that confirms that caching is indeed occurring i.e. we can see the stats that tell us how many upstream calls were made or how much data was downloaded and how much as saved by reply from cache

issue: cannot see if the cache is really working

docker run -p 5000:5000 epicwink/proxpi
Unable to find image 'epicwink/proxpi:latest' locally
latest: Pulling from epicwink/proxpi
43c4264eed91: Pull complete 
4d1b2996acac: Pull complete 
b3125c671f19: Pull complete 
c72d76b06fd3: Pull complete 
a967893e1984: Pull complete 
Digest: sha256:bf9832683416985679e1e5b6a6d2cb34bfe9c8d3ce2b07b2095ca5943242d632
Status: Downloaded newer image for epicwink/proxpi:latest
2024-11-06 16:13:56 [    INFO] proxpi.server: proxpi version: 1.2.1rc0
2024-11-06 16:13:56 [    INFO] proxpi.server: Cache: Cache(root_cache=_IndexCache('https://pypi.org/simple/', 1800), file_cache=_FileCache(5368709120, '/tmp/tmppwvz0ot8', <proxpi._cache.Session object at 0x7f01c58a49e0>), extra_caches=[])
2024-11-06 16:13:56 [    INFO] gunicorn.error: Starting gunicorn 22.0.0
2024-11-06 16:13:56 [    INFO] gunicorn.error: Listening at: http://0.0.0.0:5000 (1)
2024-11-06 16:13:56 [    INFO] gunicorn.error: Using worker: gthread
2024-11-06 16:13:56 [    INFO] gunicorn.error: Booting worker with pid: 7
2024-11-06 16:14:44 [    INFO] gunicorn.access: 192.168.88.228 "GET /index/flask/ HTTP/1.1" 200 6790 146ms
2024-11-06 16:14:44 [    INFO] gunicorn.access: 192.168.88.228 "GET /index/flask/flask-3.0.3-py3-none-any.whl.metadata HTTP/1.1" 200 0 129ms
2024-11-06 16:14:44 [    INFO] gunicorn.access: 192.168.88.228 "GET /index/werkzeug/ HTTP/1.1" 200 11875 67ms
2024-11-06 16:14:44 [    INFO] gunicorn.access: 192.168.88.228 "GET /index/werkzeug/werkzeug-3.1.2-py3-none-any.whl.metadata HTTP/1.1" 200 0 42ms
2024-11-06 16:14:44 [    INFO] gunicorn.access: 192.168.88.228 "GET /index/itsdangerous/ HTTP/1.1" 200 2374 50ms
2024-11-06 16:14:44 [    INFO] gunicorn.access: 192.168.88.228 "GET /index/itsdangerous/itsdangerous-2.2.0-py3-none-any.whl.metadata HTTP/1.1" 200 0 40ms
2024-11-06 16:14:44 [    INFO] gunicorn.access: 192.168.88.228 "GET /index/flask/flask-3.0.3-py3-none-any.whl HTTP/1.1" 200 0 69ms
2024-11-06 16:14:44 [    INFO] gunicorn.access: 192.168.88.228 "GET /index/itsdangerous/itsdangerous-2.2.0-py3-none-any.whl HTTP/1.1" 200 0 35ms
2024-11-06 16:14:45 [    INFO] gunicorn.access: 192.168.88.228 "GET /index/werkzeug/werkzeug-3.1.2-py3-none-any.whl HTTP/1.1" 200 0 103ms
2024-11-06 16:18:23 [ WARNING] gunicorn.error: Invalid request from ip=192.168.88.227: Invalid HTTP request line: '\x16\x03\x01\x07K\x01\x00\x07G\x03\x03¥\x14Ñ\x85´'

2024-11-06 16:18:44 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/flask/ HTTP/1.1" 200 6790 7ms
2024-11-06 16:18:44 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/flask/flask-3.0.3-py3-none-any.whl.metadata HTTP/1.1" 200 0 3ms
2024-11-06 16:18:44 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/werkzeug/ HTTP/1.1" 200 11875 11ms
2024-11-06 16:18:44 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/werkzeug/werkzeug-3.1.2-py3-none-any.whl.metadata HTTP/1.1" 200 0 2ms
2024-11-06 16:18:44 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/jinja2/ HTTP/1.1" 200 5876 86ms
2024-11-06 16:18:44 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/jinja2/jinja2-3.1.4-py3-none-any.whl.metadata HTTP/1.1" 200 0 65ms
2024-11-06 16:18:44 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/itsdangerous/ HTTP/1.1" 200 2374 2ms
2024-11-06 16:18:44 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/itsdangerous/itsdangerous-2.2.0-py3-none-any.whl.metadata HTTP/1.1" 200 0 2ms
2024-11-06 16:18:45 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/click/ HTTP/1.1" 200 7266 67ms
2024-11-06 16:18:45 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/click/click-8.1.7-py3-none-any.whl.metadata HTTP/1.1" 200 0 33ms
2024-11-06 16:18:45 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/blinker/ HTTP/1.1" 200 1780 63ms
2024-11-06 16:18:45 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/blinker/blinker-1.8.2-py3-none-any.whl.metadata HTTP/1.1" 200 0 42ms
2024-11-06 16:18:45 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/markupsafe/ HTTP/1.1" 200 40132 178ms
2024-11-06 16:18:45 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/markupsafe/MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata HTTP/1.1" 200 0 40ms
2024-11-06 16:18:45 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/flask/flask-3.0.3-py3-none-any.whl HTTP/1.1" 200 0 3ms
2024-11-06 16:18:45 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/blinker/blinker-1.8.2-py3-none-any.whl HTTP/1.1" 200 0 40ms
2024-11-06 16:18:45 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/click/click-8.1.7-py3-none-any.whl HTTP/1.1" 200 0 35ms
2024-11-06 16:18:45 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/itsdangerous/itsdangerous-2.2.0-py3-none-any.whl HTTP/1.1" 200 0 2ms
2024-11-06 16:18:46 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/jinja2/jinja2-3.1.4-py3-none-any.whl HTTP/1.1" 200 0 54ms
2024-11-06 16:18:46 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/werkzeug/werkzeug-3.1.2-py3-none-any.whl HTTP/1.1" 200 0 2ms
2024-11-06 16:18:46 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/markupsafe/MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl HTTP/1.1" 200 0 41ms
2024-11-06 16:18:47 [    INFO] gunicorn.access: 192.168.88.12 "GET /index/pip/ HTTP/1.1" 200 17622 71ms
2024-11-06 16:22:29 [    INFO] gunicorn.access: 172.17.0.1 "GET /index/pip/ HTTP/1.1" 200 17622 12ms

Current behaviour

Looking at say lines with "flask-3.0.3-py3-none-any" , i tried running following on 3 computers:

python3 -m pip install  --index-url=http://192.168.88.70:5000/index/ --trusted-host 192.168.88.70  flask

it worked well, but i am not sure if it is actually saving me data by serving me cached copy or if it is again proxying the request.

EpicWink commented 3 weeks ago

See https://github.com/EpicWink/proxpi/commit/f4f87a677a3ca68c67907067d80adf84f6485f2b and https://github.com/EpicWink/proxpi/commit/3cc4b78c18f8d165f206d92e9b192f4f47dd5ee0 (set environment variable PROXPI_LOGGING_LEVEL=DEBUG with latest repo check-out)

harshguptaserver commented 2 weeks ago

Nice! Thank you @EpicWink

Got it working :-) Build/run commands (for anyone who stumbles here and wants to replicate): 

 git clone https://github.com/EpicWink/proxpi.git
cd proxpi/
docker build . -t proxypi:latest
docker run -p 5000:5000  -e PROXPI_LOGGING_LEVEL=DEBUG  proxypi:latest

command on client:

python3 -m pip install  --index-url=http://192.168.88.70:5000/index/ --trusted-host 192.168.88.70  flask

service logs:

2024-11-13 14:42:31 [   DEBUG] proxpi._cache: Index 'https://pypi.org/simple/' cache stats:
flask: hits: 2, misses: 1
werkzeug: hits: 2, misses: 1
jinja2: hits: 2, misses: 1
itsdangerous: hits: 2, misses: 1
click: hits: 2, misses: 1
blinker: hits: 2, misses: 1
markupsafe: hits: 2, misses: 1
pip: hits: 0, misses: 1
2024-11-13 14:42:31 [   DEBUG] proxpi._cache: Files cache stats:
files-pythonhosted-org/packages/61/80/ffe1da13ad9300f87c93af113edd0638c75138c42a0994becfacac078c06/flask-3.0.3-py3-none-any.whl.metadata: hits: 1, misses: 1
files-pythonhosted-org/packages/52/24/ab44c871b0f07f491e5d2ad12c9bd7358e527510618cb1b803a88e986db1/werkzeug-3.1.3-py3-none-any.whl.metadata: hits: 1, misses: 1
files-pythonhosted-org/packages/31/80/3a54838c3fb461f6fec263ebf3a3a41771bd05190238de3486aae8540c36/jinja2-3.1.4-py3-none-any.whl.metadata: hits: 1, misses: 1
files-pythonhosted-org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl.metadata: hits: 1, misses: 1
files-pythonhosted-org/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl.metadata: hits: 1, misses: 1
files-pythonhosted-org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl.metadata: hits: 1, misses: 1
files-pythonhosted-org/packages/22/35/137da042dfb4720b638d2937c38a9c2df83fe32d20e8c8f3185dbfef05f7/MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata: hits: 1, misses: 1
files-pythonhosted-org/packages/61/80/ffe1da13ad9300f87c93af113edd0638c75138c42a0994becfacac078c06/flask-3.0.3-py3-none-any.whl: hits: 1, misses: 1
files-pythonhosted-org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl: hits: 1, misses: 1
files-pythonhosted-org/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl: hits: 1, misses: 1
files-pythonhosted-org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl: hits: 1, misses: 1
files-pythonhosted-org/packages/31/80/3a54838c3fb461f6fec263ebf3a3a41771bd05190238de3486aae8540c36/jinja2-3.1.4-py3-none-any.whl: hits: 1, misses: 1
files-pythonhosted-org/packages/52/24/ab44c871b0f07f491e5d2ad12c9bd7358e527510618cb1b803a88e986db1/werkzeug-3.1.3-py3-none-any.whl: hits: 1, misses: 1
files-pythonhosted-org/packages/22/35/137da042dfb4720b638d2937c38a9c2df83fe32d20e8c8f3185dbfef05f7/MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl: hits: 1, misses: 1

if possible, It would be great if we can have an endpoint to make a call and check the stats like : acng-report.html in apt-cacher-ng

Thank you for the quick response and help.