Import process

[NKAmapper]

Some thoughts about what would be needed in the next phase to get a structured import process:

• An import page is needed to get the import approved, first by the Swedish community (on the forum or the mail list) and then by the Import mail list. Requirements are set out in the Import Guidelines. Example. The import page wiki will be the "project page" for the import. It will contain for example an overview about how the NVDB attributes are converted to OSM tags.
• A structured way to keep track of status. Example
• A wiki page to provide guidelines for the import itself. Important for recruiting new participants. Example
• This Github page. Instructions for running the program (but not the import process etc).

[atorger]

Again great ideas, however I will probably not be involved in that though, not for the time being at least, my time slot working on this is sort of running out. Not everyone will agree on imports either, there's always politics, not something I particularly enjoy working with.

The tool can be used already today by experienced users for assisted manual editing. Scripted imports with automated overwriting and synching of existing data seems pretty far off to me. And as long as you just merge it manually, basically looking at each road by hand, double-checking with photos, using replace geometry tool etc, taking previous tags into account, I don't really see it as "import" as such, just a way to do manual mapping more efficiently. I just tested the tool on Dorotea Kommun (small kommun in the north). It took about 20 hours of hand-merging. About 90% of the roads where already there, of those 40% had quite poor alignment, and 10% very poor alignment. Tags like speed limits etc often wrong, missing road names. The biggest gain when making a merge like this is that you get a baseline that you can trust the quality of that roads we expect to be there are there and in the right location. There are no house addresses in NVDB though unfortunately, so that's still largely missing.

I found some improvement ideas from working with Dorotea, I hope to get time to put them in... but I cannot work as intensely as the past month any longer, so it will probably move a bit slower now for a while.

In northern Sweden we are very few active mappers and the map is still quite immature in many places, so there's not much organizing that need to be done to make improvement. In southern Sweden and densely populated areas I expect imports to be much more controversial and technical difficult, but there is also less need for them as the map is more mature.

I do support a more structured and organized way to use the tool though, but I'm probably not going to be in charge of such an effort.

[NKAmapper]

I am quite sure it will be considered an import since it is based on external data, and there will eventually be complaints and the usual threats of reverting the edits etc.

Maybe somebody else in the Swedish community could create the wiki page needed for complying with the import guidelines.

[NKAmapper]

Draft import plan: https://wiki.openstreetmap.org/wiki/Import/Catalogue/Sweden_highway_import Needs further edits.

[atorger]

Just to make clear my driving force behind this -- for my own egoistic needs, a good map in Norrbotten and Västerbotten is basically all I need, because that's where I do my cycling and use OSM end services (and have my tourism connections). In time I could cover all that personally, so I don't have a strong drive to make a nation-wide organized import at this time. But on the other hand I have nothing against it either.

I think however that it can be a bit hard, politically speaking. I'm already using this script for mapping myself and got acceptance for what I do, but so far I keep to rural municipalities where there are few other mappers and lots of outdated and missing data. My process is also very manual and quite much reworking of the output from the script, especially highway types in rural areas, adding turning circles, merging with existing roads that doesn't exist in NDVB, and fixing alignment in existing data, sometimes having to move whole villages 20 - 40 meters to be able to merge properly. Also careful to keep history and not just delete and replace (but instead use replace geometry for automated modify).

The import work should be carried out by experienced mappers, who see it not purely as an import, but a tool to map faster, with more accurate geometry and with more tags.

Thanks to my quite careful process and open discussion about what I do, my work has been accepted, but a less careful import would for sure cause great controversy and could turn the community against any type of import effort. On reason the README is in Swedish is because I don't want the script to be too wide-spread. I write English here out of habit and simply because my Norwegian just sucks, so it becomes easier to discuss issues in English for me.

There are many traditionalists in the Swedish OSM community which are very skeptical about imports, and also thinks it takes away some of the fun of mapping and originality of OSM. I understand that view, but personally I think one can argue that we need imports to maintain a decent coverage and make OSM data as useful as possible, and even if we do make imports there is lots of original on-the-ground mapping one can do in addition. However, I don't want to engage in that struggle to convince the unwilling that I expect it will be if we try to get consensus to agree that we should do a nation-wide import.

Hopefully I'm wrong, and it will be quite easy to get all on board, but I'm not the person that want to bring this out to the community.

If I would organize a strategy, I would be careful which municipalities we do, and keep away from traditionalists "hometowns" (which usually are quite well mapped anyway), and then keep an import plan just like yours above to make sure we make consistent imports, but not really try to make it a nation-wide mandate or try to get a consensus. Just be like "are you an experienced mapper and want to map more efficiently? Use this script, following these guidelines", and then it's each mapper on their own responsibility. I would not expect more than ~5 people of the Swedish community is at a level that they actually are interested in doing this kind of advanced mapping effort. If there actually would be five of us, we would cover lots of ground, fast, that would be nice for sure.

As long as we clearly improve the municipalities we map in, it's not that controversial. Problems arise when we go in and say change primary/secondary highway from OSM design to what we think NVDB mandates, and change geometry in cities like Stockholm to match NVDB despite what's already in is an alternate but still correct way to map, which does happen in cities quite often with multi-lane roads and lots of cycleways and footways.

[atorger]

I don't want to sound too negative though.

I think I'll write up a guideline which in shorter terms and more specified describes how I manually preprocess the NVDB data and merge it with existing, which then could be discussed and included on the wiki. I think a preprocessing step and manually looking at each road is key, both to getting a good result, and to get acceptance from the community.

Once we have a baseline inside the OSM database, updates will be easier to partially automate since the automation script can then get some properties from OSM and some from NVDB before looking for differences.

I'm against just putting in all tags unmodified from the script, and in order to do that change current tagging traditions. Here's a few things that need consideration:

• small road naming: often spelling mistakes, sometimes false naming gaps, lazy group naming that can be clarified
• missing turning circles, must be added manually
• residential/unclassified: often strange borders in smaller villages, leaving stubs of unclassified/residential, than needs merging
• current tradition is to map smaller skogsbilväg as track, and more important ones (popular destinations, connecting roads etc) as unclassified. NVDB cannot make that distinction between track/unclassified correctly, must be done manually. Track road quality is traditionally mapped with tracktype grade, that information does not exist in NVDB (it exists partially in the Lantmäteriet raster map, interestingly enough)
• NVDB has preliminary geometry on new skogsbilvägar, need to be corrected by matching arial photos or raster map.
• Current mapping tradition is to make roads to individual houses service (sometimes adding driveway to be explicit), if there's a group of houses make the road unclassified, unless it's grade4 or grade5 it's kept as track. In tourism, argiculture and rural areas there's a lot of this. In larger villages residential can take over the unclassified/service types.
• 50+% of bridges need position and/or size adjustment. Bridges for cycleways are often missing completely.
• Railway crossings are in my script nowadays aligned to railway geometry so they are better placed than they otherwise are, however, even the NVDB railway geometry has some offset errors in some places, so further manual tuning may be required (one can usually trust the raster map there)

Inside cities it becomes even more complex. Roads except for multilane roads are usually quite straightforward, but cycleways and footways not so much. Bridges and tunnels often needs manual fixes, quite common that a bridge over a road is connected as a crossing in the NVDB geometry. Parkings and dead-end roads have different mapping solutions. NVDB of course have no parkings as areas at all.

All these things need to be in an import wiki, and makes the import less of a pure import and more as using NVDB data as a tool to make traditional mapping quicker with smoother more accurate geometry and with more tags.

[NKAmapper]

Good idea to have a wiki for import guidelines, i.e. all the manual steps which should be checked during the import. It seems to be similar to the guidelines I have made here for Norway: Veileder Elveg-import. Could be written in Swedish, by the way (not English).

It is still the import plan which needs to be reviewed by the OSM community (according to the import guidelines), but it can refer to the additional wiki with further instructions, and the two may be reviewed together. The workflow section in the import plan could just be very basic, similar to the current state.

My main concern has been that no matter how manual the processing is, this is still an import, and it would be a pity to get reverts or to ruin one's reputation. And when making all the municipality files available, I think we take on a responsibility to not mislead anyone into an unauthorised import.

I am regarding imports like this as initiatives to just get the basics in place for OSM. Then we may continue refining that in the years to come doing what OSM is best at - local mapping of what we see on the ground. And there is no need to push for a nation wide import. After the import plan has been reviewed, we just make the files available for anyone who is interested.

[atorger]

Great, it sounds like we are in agreement. Should I start editing somewhere to contribute to this plan? My contribution would be on guidelines how to preprocess the data etc, which then can be discussed and adjusted of course.

Maybe we should align our scripts first so we agree within ourselves how we should do the tagging? I haven't had much time to look at your script's tagging yet and compare, I've added railway crossings though. There may still be things to discuss by the highway tagging, I think our current views are a bit different, I'm more into changing them manually (if there's not too much complications during merging, about 50% of my time goes to preparing the NVDB layer and 50% to actual merging), and if I understand correctly you think that there still is a possibility to use most tags unchanged from the script. But we can take that discussion in the align-scripts-thread. Now Matthias excellent work it's easy to download and see what the result is for various municipalities making it easier to discuss those issues.

(Regarding OSM as a whole I'm ambivalent which I guess shines through, I haven't had the greatest of experience of the "community" on the international level (Swedish community is fine), and I personally think the project is very poorly managed causing a stagnant technical platform and lack of quality geo services actually using the data we contribute, and with nature even blocking the possibility to contribute high quality mapping due to unwillingness to implement necessary features.

So I have a love-hate relationship and I'm not too much emotionally invested, I would not cry if this whole effort gets blocked by the community (I think that is unlikely though, Sweden desperately needs the basic map covered, which it isn't today), then I just move on to some other project. That said when I actually contribute my interest is making good mapping, so I don't think my ambivalence to OSM should be a problem in developing the script, but I don't have much respect for fundamentalist interpretations of original OSM processes and principles which I think today is a poison to the community on the international level. I haven't at all engaged in the import mailing list, I've been burnt by the tagging list already, and I plan to stay as far away from the OSM formalities I can)

[NKAmapper]

Agree, let us work through the last details of tagging before we complete the tagging section of the import plan. I have some personal preferences but no strong views about tagging. I think the most important thing is to get feedback from the Swedish community on the forum once the last details are clarified and the product is ready for a review. Now that we have all the files it should be easier to get more specific feedback.

My learning has been that if we get the tagging as right as possible (i.e. defaults which are for example 80-90% ok), the import will go a lot quicker, and there is less chance that someone takes the lazy approach and do not care about the details of tagging. I spend maybe 10% on tagging when I import, mostly adjusting highway classes between unclassified/service/track + adding turn:lanes tagging at bigger junctions.

Also, you could create a https://wiki.openstreetmap.org/wiki/Import/Catalogue/Sweden_highway_import/Import_guidelines page where you start on the guidelines. Technical details about how the script works is better to document on GitHub, I think.

I have had a number of import plans for review on the international import list, and they have been very kind every time. In fact I have a plan for review there this week which have not received a single comment: Norway Building Import.

[atorger]

Ok, I'm already so deep in this so I just created a wiki user, I'll create that import guidelines page and write some stuff which you can review later. I'll report back when I have something to show.

[matthiasfeist]

@atorger here's the page @NKAmapper already created https://wiki.openstreetmap.org/wiki/Import/Catalogue/Sweden_highway_import

[atorger]

Thanks, I'll edit that one in English, assuming the audience for that is the international OSM community as well as the Swedish, and then create a guideline wiki page "handledning" on the side in Swedish aimed at importers which will contain much of the info from the README.

[atorger]

@NKAMapper s document was already largely complete I think, I made some minor adjustments, removed the static translation table of tags and made a generic description and put a link to this script as there's like a crazy amount of static conversion to document it's just better to look at the code.

Next I'm going to work on the Swedish handledning. But I need to do some paid work too :-)

Just to be clear: I'm putting my unwillingness/ambivalence to the side now and making a push for going formal with this. I can be contact guy and the guy on the mailing lists etc. Let's give it try and see how it goes.

[atorger]

I have now made a draft of an import guideline in Swedish. It's a bit raw, but should contain "everything an importer needs to know"

https://wiki.openstreetmap.org/wiki/Import/Catalogue/Sweden_highway_import/Import_guidelines

I've also made a simple progress page:

https://wiki.openstreetmap.org/wiki/Import/Catalogue/Sweden_highway_import/Progress

Both are linked on the main page which I have updated slighty, but it's mostly NKAMapper's original work: https://wiki.openstreetmap.org/wiki/Import/Catalogue/Sweden_highway_import

So now what's next?

[atorger]

I have sent announcements to imports@openstreetmap.org and talk-se@openstreetmap.org as required by the process.

I also announced it in the Swedish facebook group and the Swedish OSM web forum.

[matthiasfeist]

Amazing! 🥳

[matthiasfeist]

does it make sense to mention my accompanying project at all those touchpoints as well? If so, I could add some things but it's not really related to the import itself.

[atorger]

Yes, I think so. The link to batch downloads would be really nice to have, have already placeholder for link. Is the server permanent enough, or do we need to find hosting somewhere? The map project is also useful and interesting in this context I think, but I was not sure if you are ready to get "heat" on it so I didn't link or talk about it yet.

FYI, there's not much activity in terms of actual reviewing going on in these lists as the "formal" part of the Swedish OSM community (those on mailing lists and to some extent OSMs own forum) has very low activity, no signs of life yet. A few comments on facebook, but that does not count in terms of the OSM process.

If no feedback is received within X days I'm not sure how to proceed, except assuming it's ok and just proceed.

[atorger]

No feedback was received, but informal approval has been given in Facebook. More discussion may appear as more people actually start using the data, but the import process part is in this context finished, so closing this issue.