enable local test develeopment without Mobility DataHub

[00Bock]

To test developments remote from Mobility DataHub/ playground, we need to set up a local test mirror from relevant databases.

[wese-da]

Sth. like:

remote: pg_dump [db] > file local: psql -f file

for relevant databases (geodata, surveyed_mobility, ...)?

[00Bock]

@dhosse So in an email you said, that you have made such a dataset. Can we share it?

[wese-da]

I'm just making a dump of the databases "surveyed_mobility" and "geodata". These should be sufficient for some tryouts.

[00Bock]

won't that be pretty big?

[wese-da]

Yes, it's a pretty big one.

Just for clarification: Is this issue related to or a duplicate of #50? Because we seem to discuss the same thing here and there, only #50 is explicitly about the development of a tool.

The way I got it now is: We want to have a local dump of modelling-relevant databases (i.e. surveyed_mobility and geodata) to 1) test local changes 2) have a showcase for customers to locally do automatic model runs

I have set up a complete db dump on my machine at the moment. Was your intention to have an excerpt of the databases? Because then, I would understand the need for a "tool".

[00Bock]

yes. It's a duplicate. But I then changed it according to your email. I just want to do 1) and 2) as you defined above...

[wese-da]

Okay. But the thing is, I made a pg_dump of "geodata" and a pg_dump of "surveyed_mobility" and set them up locally. So, I have the whole databases on my laptop. That is what I originally tried to say in my email.

But you want a subset of both databases because it's too much data to send around the globe, right?

Because in this case, we should clarify what data we want to extract for test purposes, e.g. we only take geodata from Bayern and consequently only survey data of that region or region types.

[00Bock]

how big is it? owncloud? If too big, then let's stick to osnabrück and lower saxony... sry for the confusion

[wese-da]

Just checked my owncloud memory... Should be okay, "surveyed_mobility" is about 980 MB and "geodata" around 38 GB. It may take some time to upload it, but there is enough free space. I will try to upload the dump files then, hopefully the upload size is not too small.

[00Bock]

It's 23:11... so I'll just check tomorrow!

[wese-da]

I managed to upload the surveys db into my owncloud. I had to compress it because the upload size is limited to ~ 540 MB. Which brings me to the next problem: the geodata dump is too big. Maybe I can cut the districts table to germany to reduce the size...

[00Bock]

that would be great! but, hey, no priority...

[wese-da]

The files needed for local scenario generation are now in the pg_dumps folder of my owncloud. The osm tables need to be inserted into the database via 'copy' command because they needed to be dumped as csv files and are limited to Lower Saxony because it's just too much data otherwise.

[00Bock]

works