chore: use generator in parallel mode, other misc fixes

Unstructured-IO / unstructured-api

Apache License 2.0

528 stars 110 forks source link

Closed awalker4 closed 1 year ago

awalker4 commented 1 year ago

Yield pdf splits in parallel mode to save memory
Due to this, a parallel retry test was failing. A multi page pdf will process other pages while retrying the first one, and we can't assert that our function was only hit once. The fix is to use a file with fewer pages. I copied one from unstructured.
Add a default to the minimum memory 503 check
Update pyenv instructions in readme
Add linebreaks to chunking params in readme

yuming-long commented 1 year ago

btw paddle env is not needed now (won't be an issue if paddle is not installed at all)

awalker4 commented 1 year ago

@yuming-long mind taking another look at this small pr?