Insufficient Disk Space for KartaView Downloads

[chrisdebian]

Short Description

I am using version 0.5 of MapSyncer. The script improves with each release. Unfortunately, I haven't been able to successfully download any images.Having entered my download destination, so that MapSyncer knows where to put the downloaded KartaView images, after a short wait, I am then told the size of the KartaView download, and asked whether i'd like to proceed. So far, so good. I have 250gb of local storage available, or up to 500gb of removable storage. The amount I need to download is 2164gb (2tb). The problem is that all the KartaView images have to be downloaded, before they can be uploaded to Mapilio. This means that in order for the transfer to work, I will need to buy an additional 2tb of storage.

Feature Use case

In a similar situation, when Meta/ Facebook acquired Mapillary, many people wanted to move previously captured imagery from Mapillary to KartaView. KartaView had previously been OpenStreetCam, and this seemed like a sensible move. Unfortuantely, KartaView didn't seem to be very open, and didn't care about input from the users/ contributors.

The answer to this, was for someone to write a script that prompted the user to authenticate with Mapillary and KartaView, and then the script would transfer the images, directly. By directly, I mean that nothing was downloaded to the local PC. I mentioned this a long time ago to Mapilio, but MapSyncer has been chosen as the way to import images.

So to recap, my problem is that I need 2tb of space, to download the KartaView images. Looking at Kartaview, I am currently number 69 in the world league table of uploaded images. This means that potentially, lots more people are going to have the same problem as me.

https://photos.google.com/photo/AF1QipNj_UWvgIn_t9KrL34kwq_tyrsUcajgaqWo7WB0

What are the possible solutions?

1. When the script analyses the images on KartaView, and then says how big the download of images will be, MapSyncer could check the filepath that the user has input, to save the files to, and make sure that enough local storage exists. In my case, it would say "MapSyncer is about to download 2164gb of images, but you only have 500gb of free space; do you wish to (C)ontinue, (S)elect a different location, or (Q)uit?"
2. Another option would be for me to start the KartaView download, but then escape the process (CTRL+C), when my disk was getting full. The problem with this is that because the script has stopped, it won't go on to upload my images to Mapilio. To make this better, when the script is run, it could check the filepath to see whether any image files already exist, and are ready for upload. If they are, the user could be asked if they would like to proceed with uploading the images. This would also be useful if the KartaView download was interrupted, or for example, in countries where broadband is not so resilient, the link could fail, but be resumed at a later time. If the script checked for existing local images, this could be used as a workaround for very large downloads. This possibly isn't the most elegant solution.
3. Something slightly more elegant, would be for MapSyncer to assess the KartaView download, and then ask the user which sequences of imagery they would like to download. Image sequences could then be selected (I)ndividually, in (B)atches, or (A)LL. The script could still check locally for sequences and ask if the user wanted to upload them to Mapilio.

Unfortunately, until this situation is addressed, I am not going to be able to transfer my images to Mapilio. I hope this can be sorted/ addressed.

Many thanks,

Chris chris_debian

Additional

No response

[yusufesatt]

Hi Chris,

MapSyncer is now working much better with the new update. Thank you for your contributions and suggestions.

Update the tool by typing 'pip install --upgrade mapsyncer' into the terminal and resume your uploads.

If you encounter any bugs, please feel free to report them.

Mapilio Team

[chrisdebian]

Hi,

I've tried version 0.7, but it doesn't address this issue. For people that have more KartaView images than local disk space, it seems like they have two options:

1. Proceed with the download and wait to run out of disk space, or
2. Halt the process when asked whether to proceed downloading or not.

Please let me know if this is my misunderstanding, because I would like to transfer 2 terrabytes to Mapilio, but right now, I don't think that's possible.

Thanks,

Chris

[yusufesatt]

Hi Chris,

With the last update, we now offer you how many sequences you want to download so that you can download as much as your disk space allows.

If you think you are running out of disk space, we recommend the following approach:

1. Access the sequence download screen and download sequences according to available disk space.
2. After the download is complete, you will be directed to the upload step to Mapilio.
3. After completing the upload process, remove the downloaded sequences from your disk to free up space.

When you return to the sequence download screen, you can continue the download from where you left off without any problems. Rest assured, the system will intelligently prevent re-downloading sequences that have already been uploaded to Mapilio.

We are improving our tool by listening to your suggestions and problems, please let us know the problems or suggestions you have.

[chrisdebian]

Hi, @yusufesatt

Thanks for this. I didn't see this on v0.7 after 30 mins. I'll retry and report the outcome.

Thanks,

Chris

[chrisdebian]

@yusufesatt I let the script run overnight, and am pleased to report that a list of sequences was displayed. Sadly, when I tried to download the first three sequences, I got an error to do with 'OSV' which I think was the previous name for Kartaview (OpenStreetCam > OpenStreetView > KartaView)

As an aside, how big is each sequence, just wondering how many I can download, before I run out of space?

Thanks,

Chris.

[yusufesatt]

Hi Chris,

Thank you for your feedback.

We fixed the issue as soon as possible, and it will be released in the next update.

Regarding your question about the size of each sequence, we cannot provide a clear answer as Kartaview does not have a specific number of images for sequences. However, we have added this issue to our list for future developments.

Kind regards.

[chrisdebian]

Thanks @yusufesatt . I saw your commit, I'll wait for the new release.

Cheers,

Chris

[chrisdebian]

One step closer with version 0.8. The imagery was downloaded, but I get an error, after logging into Mapilio.

For clarity, I do have a Mapilio account, but log into the app using my OSM details. So that me and others know that they are entering the correct credentials, please could you add a bit more text saying, perhaps 'Not your OSM credentials'. I'm not sure why my the script failed after authentication; could you investigate?

Many thanks,

Chris. chris_debian

[yusufesatt]

Hi Chris,

Unfortunately, there is no OSM login option at the moment, but it is included in our plans for future updates.

However, if you do not want to upload your data, you can continue the upload process by logging in with a different Mapilio account.

[chrisdebian]

@yusufesatt Hi, thanks for the information. I did use my Mapilio credentials, but thank you for clarifying.

Chris