search

ipfs / notes

IPFS Collaborative Notebook for Research
MIT License
401 stars 30 forks source link

Using IPFS in the OpenStreetMap infrastructure #427

Open RubenKelevra opened 4 years ago

RubenKelevra commented 4 years ago

I've written a ticket on the OSM operation tracker, regarding the possible use of IPFS for their mirror servers - and as an archive:

https://github.com/openstreetmap/operations/issues/388

bertrandfalguiere commented 4 years ago

For what it's worth, there is a partial mirror on IPFS mentionnend at https://wiki.openstreetmap.org/wiki/Planet.osm . It's https://ipfs.io/ipns/ipfs.placebazaar.org

RubenKelevra commented 4 years ago

Yes, I saw that. But it only contains files without any information about the date they have been added. It might be months out of date.

Apart from this, I tried to receive a file from this "mirror"... well after 10 minutes I got 4 nodes of the europe file.

I guess the mirror is long gone and someone is holding parts of the content in his cache.

bertrandfalguiere commented 4 years ago

The OSM team seems pretty hostile. I think the smart move if we want support is to maintain an IPFS mirror of OSM, and show them some numbers down the road, or to become the go-to server for IPFS users.

RangerMauve commented 4 years ago

Relevant: https://peermaps.org/

RubenKelevra commented 4 years ago

@bertrandfalguiere well I wouldn't say hostile, but not amused about the 'demand' for additional work. They are pretty stressed right now, because of high server loads because of corona induced outages and several replacements going on additionally.

I'm always worried about my tone, to native speakers ... and this time once more I get the feeling my tone was wrong. :/

I don't want to push them to do anything, but I already reserved enough space to add 1-2 redundancies in 2 European countries.

Doing this for ourselves means rsync to IPFS work again, I just wrote a script which does this, and it's not as simple as it might sound like - except if you do just full replacements every time something changes. Delta updates, on the other hand, are pretty well, shitty.

It's much easier to not parse rsync logs and do the work just while generating the files anyway.

And we would have to rely on the md5 files, which I really don't like.

RubenKelevra commented 4 years ago

Relevant: https://peermaps.org/

This uses the dat protocol, which reads like a very basic copy of IPFS 🤔

RangerMauve commented 4 years ago

Dat and IPFS are pretty different under the hood. Neither is really a copy of the other other than some basic similarities liker having p2p networking and using merkle DAGs.

The thing about peermaps is that there's a bunch of non-dat libraries in there for distributing the actual OSM data cross multiple peers and being able to process that into maps which is hard to do with the raw data unless you have some sort of superpeer.

RubenKelevra commented 4 years ago

The thing about peermaps is that there's a bunch of non-dat libraries in there for distributing the actual OSM data cross multiple peers and being able to process that into maps which is hard to do with the raw data unless you have some sort of superpeer.

Yes true. Are you part of the project? Maybe it's interesting to take a look how this would perform in an IPFS cluster?

bertrandfalguiere commented 4 years ago

There's a grant in the pipe to tackle this: https://github.com/ipfs/devgrants/pull/41

okdistribute commented 4 years ago

Hi :wave: I'm from the peermaps project. We will be pushing the files to dat, ipfs, and also hopefully filecoin.