search

open-innovations / jrf-insight

JRF North England Insight Finder
https://open-innovations.github.io/jrf-insight/
MIT License
1 stars 0 forks source link

Pull all HBAI data split by every dimension #12

Closed luke-strange closed 1 year ago

luke-strange commented 1 year ago

Pull the entire HBAI database - for each measure, have a table with every dimension.

For now we just want to pull the most recent year otherwise we get blocked by the limit of 1,000,000 cells.

luke-strange commented 1 year ago

BLOCKED by #13, so moving back into Ready for now

luke-strange commented 1 year ago

statXplore has a limit of 1,000,000 cells, so we will probably have to pull this data 1 year at a time.

luke-strange commented 1 year ago

Also want to pull last time series element for everything to get an idea of what is in the data.

luke-strange commented 1 year ago

Blocked by waiting for #39

luke-strange commented 1 year ago

well, it looks like we won't be able to pull every dimension, even when we limit to 1 year at a time. We will need to decide which dimensions we can leave out, there are probably some that won't be particularly useful for us. @ChristianSpence

ChristianSpence commented 1 year ago

Is this a volume of data to pull issue, or something else?

On Tue, 20 Jun 2023 at 11:09, luke-strange @.***> wrote:

well, it looks like we won't be able to pull every dimension, even when we limit to 1 year at a time. We will need to decide which dimensions we can leave out, there are probably some that won't be particularly useful for us. @ChristianSpence https://github.com/ChristianSpence

— Reply to this email directly, view it on GitHub https://github.com/open-innovations/jrf-insight/issues/12#issuecomment-1598492859, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJ3WPKAY4O2AOYCBBGAUADXMFZELANCNFSM6AAAAAAYFGQXVU . You are receiving this because you were mentioned.Message ID: @.***>

luke-strange commented 1 year ago

@ChristianSpence i believe its volume yes.

ChristianSpence commented 1 year ago

OK, let's discuss after the sprint planning.

On Tue, 20 Jun 2023 at 12:11, luke-strange @.***> wrote:

@ChristianSpence https://github.com/ChristianSpence i believe its volume yes.

— Reply to this email directly, view it on GitHub https://github.com/open-innovations/jrf-insight/issues/12#issuecomment-1598579705, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJ3WPIL4FQYTSXYK43XDGDXMGAOZANCNFSM6AAAAAAYFGQXVU . You are receiving this because you were mentioned.Message ID: @.***>

luke-strange commented 1 year ago

clsoing this as we have the data we need as stands. we can create a new ticket and review statxplore data pulling if/when necessary.