Android:fix SD card storage

[jkoan]

From a Facebook report i got a issue that Navit is not able to write to the SD-Card because of the new Document Provider which must be used for a

https://developer.android.com/guide/topics/providers/document-provider Also a good comment on this toppic: https://stackoverflow.com/a/26765884

---- The Log 04-26 18:50:59.579 5051 5130 E NavitMapDownloader: Could not write info file for map download. Resuming will not be possible. (/storage/0000-0000/navit/Niedersachsen.tmp.info (Permission denied)) 04-26 18:50:59.579 5051 5130 D NavitMapDownloader: size: 861307928, read: 0, timestamp: 0, Connection ref: http://maps9.navit-project.org/api/map/?bbox=6.40,51.24,11.69,54.22&timestamp=180424 04-26 18:50:59.580 5051 5130 E NavitMapDownloader: Could not open output file for writing: java.io.FileNotFoundException: /storage/0000-0000/navit/Niedersachsen.tmp (Permission denied)

[jandegr]

Do you know smth more of the used devive, api and such. Just did a test on android 6.0 downloading to SD card after giving both the permissions and changing the location to SD, all went fine

[jkoan]

https://www.facebook.com/navitproject/posts/1812419968781103 look at the comments. The log is from the same person, he''s using a samsung smartphone (Galaxy J3)

[jandegr]

I did not find the relevant entry on Facebook, but don't have a FB account either. Anyway, those 0000-0000 are weird, when I tested it was smth like /storage/C348-1DE1/navit....

[GrulClatu]

those 0000-0000 come from an original Samsung SD Card, before I had another which announced itself like 348-1DE1 - strange, but works identically. ES Dateimanager behaves on both SD cards the same like navit does.

[GrulClatu]

How ES Dateimanager reakts on my try to create a new directory at the root dir of the external SD-Card.

[GrulClatu]

Announcing that directory to navit to download maps to. Then I chose a map to download which didn't work. After that I tried to create a bookmark directory. Didn't work either. Repeated that until Navit session died.

[GrulClatu]

Almost forgot: Smartphone is a Samsung J3 with Android 7.0

[jandegr]

thx, at the end of next week I can test on a 7.1 device with an SD card

[aerostitch]

Nice. I have a Samsung J7 with a SD card & Android 7.0 (Samsung experience version 8.1), I'll try too tonight in case it's related to the samsung flavor added over Android. I'll do a session with @pgrandin to see what we can improve to make it easier to use from a user-X standpoint at the same time.

[aerostitch]

On my phone I'm able to change the location to a folder at the root of the sdcard. But when I try to download the map it says that the current map location is not available and fails to download the map. But what I noticed is that it adds "navit" at the end of the path like /storage/AEF0-1810/Navit/navit. At this point I think navit should create the missing folder. Should I create a separate issue for that?

Then I create the subfolder manually and am able to choose the map of California. From there it stays several minutes at 0% (and I'm on wifi).

Edit 1: Actually it's still at 0% after 6min.

Edit 2: still at 0% after 15min but no crash yet

Edit 3: restarted navit, start the download again, left it 10min, still at 0%.

Edit 4: tried several wifi and 4GLTE but no way to get it working. I'll move that to a separate issue later.

[jandegr]

IMHO the solution would be

• for most users, make navit use its private dir, smth like Android/data/org.navit.project AFAIK this needs no permission and at last we will have a Navit that cleans up when uninstalled, the current location does not get cleaned when uninstalled, so many users have a chunck of memory eaten away by once testing navit. For users having an SD card as extension of internal memory it stays the same. Files not accesible over USB but for 99,999% of the users there is no need to.
• the code can be modified to switch to using /storage/C348-1DE1/Android/data/org.navitproject.navit if an SD card is present and is configured as seperate storage, for the rest this works similar as above.

To have it the old way.. tested on 6.0 -Connect to a pc over usb and create a folder navit in the topdir of the SD card -For the location choose the topdir of the SD card in navit, not the navit folder since it would try to make a dir navit/navit and fail. -restart navit -now you can download a map to the SD card and it will be placed in the /navit folder you made, or you can download a map with the pc and drop it in the /navit folder you made with the pc. -I suppose this will still need the storage permission.

The last needs testing on more devices, but testing with an SD card already having a /navit folder made me believe it worked fine where tests wit a freshly formatted SD card failed.

EDIT the solution with the manually created folder fails on 7.1 this works on 7.1 (requires a code change) /storage/emulated/0/Android/data/org.navitproject.navit/files this works on 7.1 to use SD card (requires a code change) /storage/5EE3-0F1B/Android/data/org.navitproject.navit/files Both locations were retreived with getExternalFilesDirs().

Further to be investigated : files created by navit in the /navit folder are not always visible with a pc over USB, but they are always visible with the internal Android file explorer.

[GrulClatu]

Tried the old way. Created a navit folder. Tell Navit to use the root dir. Still no write permission.

[GrulClatu]

/storage/emulated/0/Android/data/ is internal sdcard (small) /storage/0000-0000 is external sdcard (big) jandegr, please ask jkoan about the api to use the android document provider!

I do have placed before manually downloaded maps to the external storage navit can read then, but if doing so, navit also wants to write bookmarks there which fails. If navit can't write bookmarks, it is almost useless, because you have to literally write each and every place you want to go.

I am really excited about navit having soon an api to the document provider which allows using the external sdcard "officially" at those android versions which have restricted write access to it. To be honest, separating maps and bookmarks storage place would be a solution, too, if bookmarks keep being at /storage/emulated/0/Android/data/org.navitproject.navit/files always. You can write bookmarks no matter whether the maps are located at internal or external sdcard and placing the map at restricted areas can be done manually if wanted.

[pgrandin]

@killerhippy

I do have placed before manually downloaded maps to the external storage navit can read then, but if doing so, navit also wants to write bookmarks there which fails.

So you can confirm that https://github.com/navit-gps/navit/issues/488 is still a valid issue?

[GrulClatu]

That depends on the Android version AFAIK. But yes, it looks like the same issue to me. Which is yours, pgrandin?

[urfin78]

Sorry for opening a new issue on this topic. For me the problem exists at least since 2015, see my forum thread here: https://forum.navit-project.org/viewtopic.php?f=17&t=564 I don't know if the information there from tryagain is still relevant:

Oh. It looks like there are another two API changes we have not worried about.

First one makes it impossible for any third party application to write to external sd card on unrooted Android 4.4 outside of a special application dir (/Android/data/org.navitproject.navit relative to sdcard root). Official documentation confirms it. We could suggest users of Android 4.4. device to select app specific directory and we should warn them that this information could be deleted (altogether with downloaded maps, bookmarks, and even configuration backups!) on app removal. Directories of the path /Android/data/org.navitproject.navit should be created by the system automatically, but sometimes that's not the case and then user would have to create them manually (on PC?).

Another one allows to overcome aforementioned limitation on Android 5.0 (and above) devices with ACTION_OPEN_DOCUMENT_TREE intent.

Also, we should notify user if map directory is not writable.

Concluding, we have some work for devs, but there seems to be a workaround: select /Android/data/org.navitproject.navit dir for map storage.

[jandegr]

Preview

Solution for Marshmallow and above

I'm doing some more testing and will publish the code for testers in a few days from now.

This shows the new settingsscreen, for devices below Marshmallow everything stays as before.

[jkoan]

@jandegr any progress? A lot of users would benefit if we would finally support this.

[GrulClatu]

@jandegr I am looking for news almost every day...

[GrulClatu]

Ping

[ChristianSchlake]

I think this issue is still present in navit 0.5.3. Are there any news?

Is navit still in development?

[lains]

So if I sum up (Reports are detailed in this comments of this Issue ticket above, but also initially here (needs to expand and display all comments)

• Issue occurs on Android 4.4 (aka Kitkat, API level 19) and higher.
• Issue occurs on devices that are not rooted.
• Steps to reproduce the issue:
  • Have an external SD card inserted in the Android device
  • Create a directory called "navit/" in the root of the SD card (otherwise set map location will refuse to select the root of the SD card, as it assumes there should be a subdirectory "navit/" in there)
  • Open navit, click on menu Tools->Set map location
  • In the file browser, select the root directory (mount point) of the external SD card
  • Exit navit
  • Re-open navit
  • Click on menu Tools->Map download
  • Try to download any new map
• When the error occurs, navit will fail downloading maps from the dialog (once a a map is selected for download, the progress bar remains at 0% and no download is done)

@jandegr has a proposal for Android 6.0 (aka Marshmallow, API level 23) and above here that could be tested.

Uploading maps directly using adb is a way work around this issue, but I can understand that this is really annoying from a user experience point of view if one is sometimes forced to use adb.

[lains]

Reproduced on another device running Android 4.4.2, and on antoher one running Android 5.1.1 (I thus editting my previous post to give an exact description on the steps i followed)

[lains]

@jandegr, can you point me to the test code you wrote back early may so that I can give it a try? Thanks.

[lains]

Changing the default location for maps in the Android FileBrowserActivity.java may also have other side effects. Recent destinations not saved anymore, changing the current layout seems persistent after quitting/restarting navit. Maybe even a crash when enabling/disabling layers was also related to changing the default location (not sure)

[lains]

Working on a solution that allows both previous behaviour (using a shared persistent directory as storage) but also proposing directories (on internal and external app-specific folders) that are compatible with Android access control (thus should allow navit to create folders and files)

[jandegr]

I recently took a little different approach, If a user installs and downloads a map without changing the path, it goes into the app's filesdir, with the advantage that if the user uninstalls navit, nothing stays behind in the device's storage guaranteed. Upgrade preserves the maps. On versions before the dynamic permissions nothing else changed so the user can change the location just as before. On more recent versions it does not ask for permission storage anymore, no need to bother the user if he's not going to change the location anyway. Users wanting to fiddle with the location on those versions are supposed to know what they are doing and must give permission manually.

But instead of using the dedicated screen I showed earlier that proposed the possible storage media, there is also the option to tweak the filebrowser activity to show those options

[lains]

OK, then I let you go ahead with the fix. I did not implement much of the user interface itself, as I am not very familiar with Android GUI. Here is a few lines I have put together for debug in order to test the various path where data can be stored:

    /**
     * @brief Get an array of strings containing canonical paths to directories where we can store data
     *
     * @note The path are expressed in their canonical form
     *
     * @return An array containing path to directories
    **/
    public String[] getPossibleStoragePath() {
        ArrayList<String> result = new ArrayList<String>();  /* The list used to build the result array */
        try {
            String sharedPersistentPath = Environment.getExternalStorageDirectory().getCanonicalPath();
            Log.d(TAG, sharedPersistentPath + " is the default shared persistent directory");
            result.add(sharedPersistentPath);
        }
        catch (IOException e) {
        }

        String mountedSDCardPath = System.getenv("SECONDARY_STORAGE");
        if ((mountedSDCardPath == null) || (mountedSDCardPath.length() == 0)) {
            mountedSDCardPath = System.getenv("EXTERNAL_SDCARD_STORAGE");
        }

        Log.d(TAG, "Guessed SD card path: \"" + mountedSDCardPath + "\"");
        File[] possibleStorageDirList = ContextCompat.getExternalFilesDirs(getApplicationContext(), null);

        if (possibleStorageDirList != null) {
            /* Secondary storage is know using env variable $SECONDARY_STORAGE */
            for (int i = 0; i < possibleStorageDirList.length; i++) {
                try {
                    String currentDirPath = possibleStorageDirList[i].getCanonicalPath();
                    /* Change to canonical */
                    if (mountedSDCardPath != null && mountedSDCardPath.length() != 0 && currentDirPath.startsWith(mountedSDCardPath)) {
                        /* This currentDirPath corresponds to an external SD card storage */
                        Log.d(TAG, currentDirPath + " is on an external storage");
                    } else {
                        Log.d(TAG, currentDirPath + " is on an internal storage");
                    }
                    result.add(currentDirPath);
                }
                catch (IOException e) {
                }
            }
        }
        String[] resultArray = new String[result.size()];
        result.toArray(resultArray);
        return resultArray;
    }

[jandegr]

Hi I have been occupied with reworking the existing code a bit so I did not do furhter work on this issue but if I had to work on it today I would start from the following thoughts

after #877 the path defaults to getApplicationContext().getFilesDir() and that is always the onboard storage then, ContextCompat.getExternalFilesDirs gives them all, incl. the onboard storage, so the extra one(s) you got is the removable one.

On devices with marshmallow and up, then only the storage volum can be selected, no custom paths anymore.

If a user did not change to the removable storage, then allow for download of binfiles over 3.8 GB. If a user changed to the removable storage, then the download limit stays.

In case a user added a fast SD card as expansion (the best solution IMHO) instead of a seperate volume, this is not reported seperatly and a download limit would not apply either.

I have briefly tested downloading binfiles > 3.8 GB but have not tested that yet in a setup with an SD card as expansion. I somebody has such a setup I would be interested in the ouctome of such a test. (positive result is expected)

[jandegr]

Hi, While cleaning up some old branches I stumbled onto the lines I used to test this last year. I posted them in https://github.com/navit-gps/navit/pull/908

[lains]

I have briefly tested downloading binfiles > 3.8 GB but have not tested that yet in a setup with an SD card as expansion. I somebody has such a setup I would be interested in the ouctome of such a test. (positive result is expected)

This is something I could test, but I guess that SD card must be not formatted as fat32 because of the 4GB limit.

[jandegr]

I have briefly tested downloading binfiles > 3.8 GB

Tested a bit more, with the predefined area of europe from the mapserver this time, good for 15053MB, first test went completely fine.