search

InfoSecInnovations / concierge

Repo for Concierge AI dev work
Apache License 2.0
174 stars 31 forks source link

Parallel processing - loading #2

Open besimorhino opened 8 months ago

besimorhino commented 8 months ago

Parallel processing should be explored for the loaders. The current flow is single threaded and results in a single CPU thread quickly hitting 100% while other threads remain under utilized.

Potential suggestion: redo loaders to have default behavior where if loading is happening, but CPU utilization is below 80%, start another loader thread/process.

besimorhino commented 8 months ago

May use Gnu parallel or PyPy (or a combo of both)

besimorhino commented 7 months ago

Investigations show that best case will be with: asyncio https://docs.python.org/3/library/asyncio.html