Page split logic adjustment

Unstructured-IO / unstructured-js-client

A Typescript client for the Unstructured hosted API

MIT License

40 stars 12 forks source link

Page split logic adjustment #62

Closed mpolomdeepsense closed 5 months ago

mpolomdeepsense commented 6 months ago

[x] ~~the default for pdf_split_page is set to True~~ the default for pdf_split_page is set to False
[x] introduced contants for controlling split size: MIN_PAGES_PER_THREAD and MAX_PAGES_PER_THREAD
[x] number of threads used for processing each PDF split can be configured through split_pdf_threads parameter instead of env var
[x] evenly distributed split sizes
[x] basic tests for the logic mentioned above

mpolomdeepsense commented 6 months ago

@awalker4

Ready for review, I think publishing is disabled for js-client, would probably need to enable it after the merge.

mpolomdeepsense commented 6 months ago

Only important changes are in: SplitPdfHook.ts, client_overlay.yaml and inside test directory.

I have also enabled CI for unit and integration tests.

Also modified speakeasy workflow to include overlay during client generation.

mpolomdeepsense commented 5 months ago

@awalker4

Not sure if I should merge it right now, the api hasn't been updated yet with the new version of the openapi spec that Hubert fixed. That means the generated client won't be working properly.

There is also a matter of publishing the new version of the client, I believe it's disabled so it won't be published automatically.

awalker4 commented 5 months ago

Yeah, since we aren't publishing yet id say go for it and then we'll get the openapi fix deployed soon