Fail gracefully when parsing takes too long - Githubissues

etalab / csvapi

An instant JSON API for your CSV

MIT License

31 stars 6 forks source link

Fail gracefully when parsing takes too long #81

Open abulte opened 2 years ago

abulte commented 2 years ago

Cf https://github.com/etalab/csvapi/issues/80

abulte commented 2 years ago

File included for reproduction: data.gouv-mars2022.xlsx

After a little research:

we can't rely on asyncio.wait_for to wrap the parsing with a timeout (probably because the parsing blocks everything)
the underlying library of agate-excel (openpyxl) has not trouble opening the file, hanging happens when converting to agate.Table
we could try to wrap the synchronous part of parsing in a timeout, but this would involve threads in an async env (mayhem)
maybe limit based on number of columns detected?