search

hasadna / datacity-k8s

1 stars 0 forks source link

Hasadna instance max upload size #127

Closed akariv closed 3 years ago

akariv commented 3 years ago

This task keeps failing, and I suspect that it's related to the max upload size for resources. The size of the file I'm trying to upload is 1,359,926,398 bytes... Can you confirm?

https://ckan-dgp.datacity.org.il/en/status/budgetkey-contract-spending-5fa9267a

OriHoch commented 3 years ago

at the moment it seems max upload size is 100mb

I can increase it, but I think having such large datasets is a big strain on the systems, especially the datastore and the automation tasks, so maybe we can allow it but limit the processing of these files

akariv commented 3 years ago

I also saw that 100mb limit but on the other hand, I did upload a 327mb file so not sure if that's the correct number.

Some of the hasadna datasets are going to get quite big (especially if we're going to bring data from 250 municipalities), so I think we should find the way to work with these large files which also maintains their usability.

On Wed, Nov 11, 2020 at 2:26 PM Ori Hoch notifications@github.com wrote:

at the moment it seems max upload size is 100mb

I can increase it, but I think having such large datasets is a big strain on the systems, especially the datastore and the automation tasks, so maybe we can allow it but limit the processing of these files

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hasadna/datacity-k8s/issues/127#issuecomment-725395026, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACAY5J6Q5SYXVDJ6RT6PXTSPJ7HTANCNFSM4TPTGE2A .

niryariv commented 3 years ago

@akariv wouldn't the budget files be separate per municipality?

akariv commented 3 years ago

Probably not, as we want to aggregate them to be useful.

niryariv commented 3 years ago

continued in #131