search

pythonspeed / filprofiler

A Python memory profiler for data processing and scientific computing applications
https://pythonspeed.com/products/filmemoryprofiler/
Apache License 2.0
837 stars 24 forks source link

ARM wheels for Linux #395

Closed itamarst closed 2 years ago

itamarst commented 2 years ago

Requested by @cblakkan, and no doubt useful to others, this is good for running on Docker.

cblakkan commented 2 years ago

Just checking back in on this to see if it works yet, any eta for when you might be able to do this?

#0 24.34 ERROR: Could not find a version that satisfies the requirement filprofiler==2022.7.1 (from versions: none)
#0 24.34 ERROR: No matching distribution found for filprofiler==2022.7.1
#0 24.34
#0 24.34 [notice] A new release of pip available: 22.2 -> 22.2.2
#0 24.34 [notice] To update, run: pip install --upgrade pip
#0 24.45 Traceback (most recent call last):
#0 24.45   File "/venv/bin/pip-sync", line 8, in <module>
#0 24.45     sys.exit(cli())
#0 24.45   File "/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
#0 24.45     return self.main(*args, **kwargs)
#0 24.45   File "/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
#0 24.45     rv = self.invoke(ctx)
#0 24.45   File "/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
#0 24.45     return ctx.invoke(self.callback, **ctx.params)
#0 24.45   File "/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
#0 24.45     return __callback(*args, **kwargs)
#0 24.45   File "/venv/lib/python3.10/site-packages/piptools/scripts/sync.py", line 175, in cli
#0 24.45     sync.sync(
#0 24.45   File "/venv/lib/python3.10/site-packages/piptools/sync.py", line 248, in sync
#0 24.45     run(  # nosec
#0 24.45   File "/usr/local/lib/python3.10/subprocess.py", line 524, in run
#0 24.45     raise CalledProcessError(retcode, process.args,
#0 24.45 subprocess.CalledProcessError: Command '['/venv/bin/python3', '-m', 'pip', 'install', '-r', '/tmp/tmp4v93a9q6', '--index-url', 'https://artifactory.local.23andme.net/artifactory/api/pypi/pypi/simple']' returned non-zero exit status 1.
itamarst commented 2 years ago

Almost got there and then ran out of time to fight CI. Maybe I'll do a YOLO version after some manual testing, worst case.

itamarst commented 2 years ago

OK, in theory this is working now in main, so hopefully should get this done in next few days.

itamarst commented 2 years ago

Presumably a couple more typos somewhere to fight, but really getting very close. It's a bit slow insofar as CI takes multiple hours to build ARM wheels, so I only check back once a day...

itamarst commented 2 years ago

OK, these are now available. Please let me know if they don't work.

cblakkan commented 2 years ago

Appears to work perfectly, much appreciated!

itamarst commented 2 years ago

Excellent! Let me know if you have any other issues or feature requests.

You may also be interested in my other profiling project, Sciagraph, which will give you both memory and performance profiling for code running in production. Fil has a significant performance hit because it's tracking everything, which is fine for development but not really a good idea for production. Sciagraph uses sampling, which is less accurate for small scale dev experiments, but works fine in practice once memory usage is large, i.e. in production. And you get performance profiling too.

The big picture idea is you always have profiling on by default, and when something is slow or uses too much memory, you have an already-created profiling report from the most realistic of environments, production. More details at https://sciagraph.com

(There's no Metaflow integration yet, which I believe is what you all use? But for first few customers I've been including implementation of open source integrations as needed, along with additional help.)