Allow to upload .osm files to the wiki

[johsin18]

I would like to suggest to allow uploading .osm files to the OpenStreetMap wiki, in addition to pictures and LibreOffice documents. This would come in handy when proposing imports, such as https://wiki.openstreetmap.org/wiki/Canton_Basel-Stadt_Building_and_Address_Import where I had to place links to my website as a workaround. IMHO it would be better if the Wiki was self-contained, otherwise suffering from dead links sooner or later.

For to be imported data, also ZIP archives might serve well, but at least .osm files feels to me like a very natural addition. A size limit would have to be imposed, of course, for my purposes, even a limit as low as 1MB would be fine.

[h3lix1]

The change was pulled in today. Let me know if it works as expected.

A MIME type might be necessary if it tries to download .osm as text. My wiki access is restricted from uploading, so I can't test downloading. Please update with results of your testing.

[johsin18]

Thanks for the support.

However, I still cannot upload the following file: http://www.jsingler.de/osm/Landauerhof.osm (using Chrome 64 on Windows 10) The error message is File extension ".osm" does not match the detected MIME type of the file (application/xml).

And indeed, .osm files are .xml files.

[h3lix1]

I thought MIME types would be an issue, but I was thinking downloads rather than uploads.

Thanks for the heads-up. I'm working on the configuration to include application/xml for .osm

[h3lix1]

johnsin18 please try again. I give it a 50/50 shot this will work. If it doesn't, please submit the error message returned. There is one additional file that is left that may need an update, but I am unable to verify.

Also, if you are part of the admins group, let me know if you can add me to the Autoconfirmed Users group so I can test these file uploads.

[johsin18]

It still fails, giving the very same error message.

I don't understand why you cannot test this. Just download the sample file. I am a normal OSM user without any special permissions whatsoever. Anybody should be able to create an OSM wiki account like mine.

[johsin18]

Even when I check "Ignore any warnings", it still fails.

[johsin18]

How can I help to get this fixed?

[Tigerfell]

The link between the MIME type application/XML and the .osm file extension is missing. I successfully tested a change in the settings (based on the relevant handbook page). However, this allows uploading all files with the correct MIME type and extension (not limited to OSM files).

This brings up the security aspect. I tried to figure out if uploading random XML files is a security issue. There is a ticket at Wikimedia about the requirements for file types of uploads (https://phabricator.wikimedia.org/T90385#1060543). The interesting words are "allows us to disable the parsing of those, like we do for xml-based formats". I guess they refer to this section from the documentation: https://www.mediawiki.org/wiki/Manual:Security#Upload_security.

Is uploading random XML a security issue? Does disabling parsing of XML provide adequate security?

[johsin18]

No parsing is needed, from my side. And without parsing, I cannot imagine this being a security risk.

[Tigerfell]

How about the readers who will open the file? Is there any chance to sneak in some code that will be executed?

[mmd-osm]

Of course you can do stupid stuff if you allow everyone to upload some random XML which later gets rendered by a browser as XML. See https://en.wikipedia.org/wiki/Billion_laughs_attack for some inspiration.

I don't really see a use case for this sort of data. Just create some git* repo and link to it on your wiki page, and you're done. What's the benefit of uploading such data in the wiki, which won't really work for large and/or many files anyway.

[Tigerfell]

@pnorman @Firefishy Maybe the maintainers can make a decision? (I do not want to ping Tom Hughes all the time.) Either we allow uploading XML or we disallow it. In the first case, a PR is needed, in the second case we need to undo openstreetmap/chef@846f07eb77ed692c8b295cc34e526891572c9432 (which does not really do anything yet).

[Firefishy]

I am not excited by the idea of the wiki becoming cloud storage for random .osm files. I am unsure what the issue currently is. 1) We added them as a supported file but people cannot upload them because of a mediawiki security issue? or 2) People cannot download them because of some mime-type issue?

[tomhughes]

I think the issue is that by default MediaWiki tries to get the mime type of any uploaded file and match it against $wgMimeTypeBlacklist and if it's not able to determine a mime type then it fails that check and considers it blacklisted.

The MediaWiki documentation seems to suggest that the type determination is entirely based on the extension, which seems a bit pointless as other people here have said, because you can just change the name of the file... But the php module it refers to seems to suggest is uses a magic based (ie file contents matching) system.

[Tigerfell]

There are two issues

• Technical: The system does not allow the upload of XML with .osm file extension, because it first checks if the extension is allowed (it is allowed as configured in openstreetmap/chef@846f07eb77ed692c8b295cc34e526891572c9432) and secondly it checks the MIME type which is "application/XML" (determined by the internal "magic") and uses a list to match extension and MIME type. The last step fails because the entry is missing (basically you declare "osm" a valid extension for "application/XML").
• Security: Apart from that I raised the security issue, because MediaWiki just detects the MIME type and the file name extension. So we would effectively allow all XML uploads with the file name extension "osm". I was wondering if it would be possible to upload a malicious file which would be executed within the system. Apparently it is possible to disable the execution of XML files in the webserver (https://www.mediawiki.org/wiki/Manual:Security#Upload_security). Additionally, I was wondering whether there is a possible security problem if a user just downloads a previously uploaded XML file (yes as stated by mmd-osm).

After allowing osm files, MediaWiki issues the following warning when previewing a file:

Warning: This file type may contain malicious code. By executing it, your system may be compromised.

This is the same you get after uploading ODF (see https://wiki.openstreetmap.org/wiki/File:Radrevier_ruhr_korrekturen.odt for example).

I would like to hear your opinion regarding this warning. Is this a reasonable risk (I mean we currently allow ODF)? Are you confident that disabling the execution of XML files within MediaWiki is sufficient?

[tomhughes]

That description does not appear to match the MediaWiki documentation.

There is no mention of a whitelist for mime types, only a blacklist. As I say I suspect the real problem is that unknown mime types (ie extensions which do not map to a mime type) are implicitly rejected because it would not be able to match them against the blacklist.

Certainly matching it to application/xml would be a terrible idea - something like application/x-openstreetmap+xml might be appropriate but if it's done just on the extension then it's no real security at all.

I am also unconvinced by your statement about how MediaWiki works, though I accept you are just repeating what they say. The documentation for the PHP module it uses (https://www.php.net/manual/en/function.mime-content-type.php) however says that magic.mime is used which implies actual fingerprinting of the file contents. It may fall back to name based matching on mime.types if that fails of course.

[Tigerfell]

The MIME type is detected as application/XML as stated previously, but I think we try to say the same thing: MediaWiki can not match type and extension and therefore rejects the file. The MediaWiki documentation suggests to use mime.types for matching (https://www.mediawiki.org/wiki/Manual:MIME_type_detection#MIME_type_validation). So magic.mime must have failed before, I guess?

Is it possible to match the files with some other MIME type if it already detects application/XML? If that was true, we could write a definition for OSM XML to narrow down the MIME type.

[tomhughes]

Ah I think I may have misunderstood. I thought that $wgFileExtensions was matched against the filename from the client but if I understand correctly the actual procedure is:

1. uploaded file is sniffed via magic.mime to get a mime type
2. mime type is matched in mime.types to get an extension
3. extension is matched against $wgFileExtensions

Which is like the most convoluted approach possible... Why not just have the white list match the mime types directly...

Anyway in that case the only way we could even think about doing it would be to add a new magic match to match as application/x-openstreetmap+xml or something and then a mime type entry to map that to .osm for the white list.

But that's a lot work for something where, as @Firefishy says, there really doesn't seem to be a good use case.

[johsin18]

1. I have given a very valid use case in my request. After all, it's required to document imports and present sample changes. Hosting the files somewhere else violates self-containedness.
2. "becoming cloud storage for random .osm files" WTF? I mean, the whole Wiki is based on trust and common sense, right? Files that are not linked to a page will be listed in the respective "Special Page". Feel free to remove them.
3. The already allowed file types include; odt, ogg, flac, and doc. Aren't those much more dangerous (e.g. macro viruses)? And what use has lossless audio for OSM? And odt files are also XML, just zip-compressed, aren't they?

[Tigerfell]

From the perspective of a wiki user:

1. The request is not closed, it was just stated that the investment to benefit ratio is pretty low. In a lot of cases, links to overpass are used.
2. There is the problem that the wiki also hosts files for the forum, the mailing list, and archived proposals. All three of them are not considered in the file usage and the special page.

[Firefishy]

I have looked into this... It doesn't look to me like Mediawiki supports setting a custom mime-type.

Someone has been able to upload an osm file to the wiki: https://wiki.openstreetmap.org/wiki/File:OSM_Heart_Embroidery_Logo_5cm_in_JEF_format.osm https://wiki.openstreetmap.org/wiki/File:Osm-heart.DST.osm.osm

But they appear not to be OSM XML files.

There likely needs to be some logic to fix: https://github.com/wikimedia/mediawiki/blob/master/includes/libs/mime/MimeAnalyzer.php

[Firefishy]

Reference 2 changes https://github.com/openstreetmap/chef/commit/846f07eb77ed692c8b295cc34e526891572c9432 and https://github.com/openstreetmap/chef/commit/583a01cf4a834db5e803b7004e5b7bd03284e7b6