Webdav Support

[rado0x54]

Resolves #148 and #188

Note, this PR includes #205

feat: Webdav support for remote storage

- Created common interface that GDrive and Webdav implement (rfs::IRemoteFS)
- Moved shared functionality into shared interface/implementation
- drive.h/.cpp was replaced by remote.h/.cpp
- fld.cpp now gets a copy of RfsItem (former gdIems), because the implementation is not required to retain these (e.g. Webdav does not)
- UI presentation changed from [GD] to [R] (Remote)
- Documentation

Furthermore the PR contains

• feat: Added CMakeLists.txt to support Jetbrains IDE (not for building)
• aligned all SDL2 imports to use <SDL2/*>
• I've left some TODOs in the gd.h interface. These only concern initialization behavior (driveInit). Happy to support fixing those in a later PR.

[ShadwDrgn]

this PR does not compile for me. Perhaps i did something wrong. HEre's what i did:

docker run --rm -ti -v $PWD:/app devkitpro/devkita64

Inside Container:

dkp-pacman -S switch-curl switch-freetype switch-libjpeg-turbo switch-libjson-c switch-libpng switch-libwebp switch-sdl2 switch-sdl2_gfx switch-sdl2_image switch-zlib switch-tinyxml2
cd ~
git clone https://github.com/J-D-K/JKSV.git
cd JKSV
make

The above compiles fine. If I then run this:

git fetch origin pull/207/head
git checkout -b pullrequest FETCH_HEAD
make

this time i get the errors in the below pastebin

output: https://pastebin.com/kKs0z0De

[rado0x54]

@ShadwDrgn Thanks for the feedback. Not sure about this error but I'm happy to try to reproduce it on my end. For now, here is a release build and here are the instructions.

edit: Please note, this Release Build is not up-to-date anymore. Please build it yourself from this branch.

[impeeza]

Normally for JKSV build, on a recently created MSYS2 environment I need to add this packages:

pacman -Syu --needed --noconfirm git make subversion base-devel mercurial python3 unzip mingw-w64-x86_64-cmake mingw-w64-x86_64-gcc mingw-w64-x86_64-glew mingw-w64-x86_64-SDL2 mingw-w64-x86_64-toolchain mingw-w64-x86_64-SDL switch-dev devkitA64 dkp-toolchain-vars libnx switch-tools switch-mesa switch-libdrm_nouveau switch-sdl2 switch-freetype switch-libpng switch-zlib switch-libjpeg-turbo switch-curl switch-libjson-c switch-sdl2_gfx switch-sdl2_image

and now for the PR#207 the package switch-tinyxml2 also was needed. after install all these packages and fetch the PR 207 the build goes ok:

On summary I did ran this commands on a recently created MSYS2 DevKitPro installation:

pacman -Syu --needed --noconfirm git make subversion base-devel mercurial python3 unzip mingw-w64-x86_64-cmake mingw-w64-x86_64-gcc mingw-w64-x86_64-glew mingw-w64-x86_64-SDL2 mingw-w64-x86_64-toolchain mingw-w64-x86_64-SDL switch-dev devkitA64 dkp-toolchain-vars libnx switch-tools switch-mesa switch-libdrm_nouveau switch-sdl2 switch-freetype switch-libpng switch-zlib switch-libjpeg-turbo switch-curl switch-libjson-c switch-sdl2_gfx switch-sdl2_image switch-tinyxml2
git clone --recursive https://github.com/J-D-K/JKSV.git
cd ~/JKSV
git fetch origin pull/207/head
git checkout -b pullrequest FETCH_HEAD
make -j

And the PR207 is build without problem, will test this night on the console.

[ShadwDrgn]

Updated my comment above to be more specific to reproduce compile failures. Dunno why this would compile fine on MSYS2 but not the official devkitpro docker image.

[impeeza]

may be a missing package on the docker or an incompatible version from Windows - Linux I did use MSYS2 on windows 11

[rado0x54]

Thanks @impeeza. Indeed, xml parsing is required for the Webdav response. I've updated the build instructions to add switch-tinyxml2.

[ShadwDrgn]

Thanks @impeeza. Indeed, xml parsing is required for the Webdav response. I've updated the build instructions to add switch-tinyxml2.

actually i did install switch-tinyxml2 but still same errors.

[rado0x54]

@ShadwDrgn Interesting. I can reproduce the error in the container. Funnily enough it builds fine on my system. I'll look into it.

[rado0x54]

Ok, @ShadwDrgn, I found your error. It's funny because my branch did it more right, but master was hiding the problem.

Base problem is that gd.h exists outside of the project and was included first

. /opt/devkitpro/portlibs/switch/include/gd.h

The problem did not show up, because before there is a unnessary recurive include via fs.h

. /app/inc/fs.h
[...]
.. /app/inc/fs/drive.h
... /app/inc/fs/../gd.h

I'll fix it in my branch. Please retry.

[ShadwDrgn]

nice. compiles now. Will see if i can test it this weekend. honestly i don't use my switch much anymore cuz i got a steam deck, but will be nice to have a simple way to get my saves off my switch to my deck quickly.

[impeeza]

Can confirm using the latest build works fine, I just finishing to setting up a WebDav service on my Windows 11 and then create the needed JSON file and test to upload a backup to the WebDav server, I don't know why the download saves from WebDav is not working Will do more test because could be my fault on setting up the WebDav server.

[zzeking]

Thanks a lot bro. JKSV actually shows WebDAV started successfully. I have successfully uploaded my saves to my WebDAV server. And if I have some methods to upload all saves by one click？

[zzeking]

Can confirm using the latest build works fine, I just finishing to setting up a WebDav service on my Windows 11 and then create the needed JSON file and test to upload a backup to the WebDav server, I don't know why the download saves from WebDav is not working Will do more test because could be my fault on setting up the WebDav server.

I also stuck in downloading

[rado0x54]

Hey I used this docker image for development and testing and now also for my online sync setup. In order to debug any download problems I would ideally need, any information from the local logs (/JKSV/log.txt) or server side access or error logs that tell me what is going on. I can also only work on this on the weekend in a limited capacity.

And if I have some methods to upload all saves by one click?

The UI/UX is the same as the GD implementation and was not changed in this PR. This can be addressed in a different PR.

[zzeking]

Hey I used this docker image for development and testing and now also for my online sync setup. In order to debug any download problems I would ideally need, any information from the local logs (/JKSV/log.txt) or server side access or error logs that tell me what is going on. I can also only work on this on the weekend in a limited capacity.

And if I have some methods to upload all saves by one click?

The UI/UX is the same as the GD implementation and was not changed in this PR. This can be addressed in a different PR.

I think it upload the save successfully, but when I download the save that the logs show me 404 error.

the jksv log is here.

[rado0x54]

Thanks @zzeking. I've pushed a fix to what I think it is. If the problem still arises, please send me the output of the following CURL against your server.

curl -v -i -X PROPFIND http://server/path/JKSV -H "Depth: 1" -u username:password

Thanks.

[impeeza]

Hello there, I was on my way testing yesterday version and found you put a commit, quickly I build a new NRO with the latest version and can confirm what with my WebDav (IIS) on Windows 11 now uploading and downloading saves from it works fine.

if you like I can write a small text about setting up a WebDav server on windows 11, let me know.

[zzeking]

@impeeza can you give your nro file to me. the new version i compiled can't even find the uploaded save (in jksv with "[R]" mark).

[impeeza]

@impeeza can you give your nro file to me. the new version i compiled can't even find the uploaded save (in jksv with "[R]" mark).

Of course buddy, here is it: 2023_07_11-6b8c3da1-PR207-JKSV.zip

[zzeking]

@impeeza can you give your nro file to me. the new version i compiled can't even find the uploaded save (in jksv with "[R]" mark).

Of course buddy, here is it: 2023_07_11-6b8c3da1-PR207-JKSV.zip

Thanks a lot, but It didn't works fine on my devices

[impeeza]

how can I help you?

[impeeza]

Hello there, I just build a new NRO using [1aca15a] Works fine on my console and a WebDav Server on Windows 11 (not using HTTPS), The only thing what could be improved is what the setting "Auto-upload to Drive/Webdav:" is not working, I enabled it but no new backup is auto uploaded. Manually uploading and downloading backups works like a charm.

By the way, I have an updated /romfs/lang/es-419.txt file I don't know how to add to your PR, but if you like you can use it:

[impeeza]

I just used the Export language files from inside JKSV and compare with the file I did edit, and filled the missing translations, so I did create this new revision.

es-419_2023_02_23-1aca15ae-PR207-V2.txt

[zzeking]

Thanks @zzeking. I've pushed a fix to what I think it is. If the problem still arises, please send me the output of the following CURL against your server.

curl -v -i -X PROPFIND http://server/path/JKSV -H "Depth: 1" -u username:password

Thanks.

Hello there, I build the newest commit. It's still going wrong.

And this pic is my webdav.json:

this is the command I typed(curl).

And I get the output below(curl):

Upload successfully but fail to delete(server logs).

Also fail to download(server logs).

[rado0x54]

Hey @zzeking your config is wrong. It should be:

origin: https://<>.top basePath: dav

please retry with that. I take it as feedback that maybe I should join these back up in the config and separate it in code myself. But for now, with the right config, it should work.

[rado0x54]

The only thing what could be improved is what the setting "Auto-upload to Drive/Webdav:" is not working.

Yeah, as mentioned I did not change anything around the UX from the GD implementation. I'd also like that feature, so I probably just put it in this PR when I find the time.

[rado0x54]

@zzeking All good with the new config? @J-D-K Let me know how I can support to get this merged.

[impeeza]

Hello there a update to my setup, in order to make easiest to setup a WebDav server, I try to start a Apache 24 on my computer, disabled the IIS setup and start over on Apache.

All went fine, I did test the WebDav first using a Ubuntu machine and can connect via the URLs: dav://192.168.100.8/Entrada and davs://192.168.100.8/Entrada Without trouble: even accessing from a web browser the folders can be listed:

But on JKSV the folders do not work just fine. for the game the saves on the WebDav folder are enumerated but without name, and even you can upload a save from JKSV backup but again is listed as remote without name: at first connection after upload a savegame:

But if I enable again IIS WebDav setup all works fine:

the only difference I see is the format when you access via Web browser, but that shouldn't matter because the WebDav protocol doesn't use the generated web page (I think so)

when using iis you get a different page:

Any idea? anything I need to config on the Apache server to work with JKSV on WebDav mode? Thanks guys.

[rado0x54]

Hello @impeeza, please send me a curl output (as described here for both the Apache2 and IIS. Maybe we see something that they do different. Please make sure you do that for a Game directory that contains the actual backups.

Furthermore I'm not familiar with the dav and davs prefixes, I hope they just get converted to the proper http(s) ones, so your curl should probably use that. Thanks

[impeeza]

Hello there, dav and davs are the prefix to use WebDav protocol on Ubuntu file manager.

Here are the outputs for Apache and IIS SalidaApache.txt SalidaIIS.txt Thanks a lot for your help.

I use Ubuntu file manager and Total commander to test WebDav access. On both WebDav and WebDav over HTTPS works fine, with either Apache or IIS server.

Let me to know how else can I do to help.

[rado0x54]

Well, the apache output is missing the <D:displayname>, which explains your behavior. Maybe you can check if you can get this enabled on Apache?

Just to make sure, the functionality still works, you just don't see the name, right?

Edit: I don't see any option of mod_dav supporting this, so I'll have to code a fallback for this scenario. I'll put it as a TODO.

Edit2: I use nginx-based webdav (via ugeek/webdav), if you want to try an alternative until then.

[rado0x54]

Looking at the response a little more. Apache is doing a really dumb thing with declaring the same namespace twice (xmlns:D and xmlns:lp1) and then using both in the subsequent response. The JKSV is definitely NOT robust to that.

I think my personal recommendation (if possible) is to use another Webdav server for now, because it's a number of issues that need to be addressed.

<D:multistatus
    xmlns:D="DAV:">
    <D:response
        xmlns:lp2="http://apache.org/dav/props/"
        xmlns:lp1="DAV:">

[impeeza]

Looking at the response a little more. Apache is doing a really dumb thing with declaring the same namespace twice (xmlns:D and xmlns:lp1) and then using both in the subsequent response. The JKSV is definitely NOT robust to that.

I think my personal recommendation (if possible) is to use another Webdav server for now, because it's a number of issues that need to be addressed.

<D:multistatus
  xmlns:D="DAV:">
  <D:response
      xmlns:lp2="http://apache.org/dav/props/"
      xmlns:lp1="DAV:">

Well even the good ones make dumb things sometimes, well as far so now will leave the IIS as server, the goal is use a easy to setup server, IIS can be cumbersome even so difficult sometimes.

Will wait for some advance on the issue, Thanks a lot for your help.

a additional thought, will be possible to upload uncompressed savefiles to WebDav server? so I can setup synchronize saves between Yuzu on my PC and the console, I know is a lot to ask.

[impeeza]

the uncompressed upload has been mentioned on https://github.com/J-D-K/JKSV/issues/214

[GrayXu]

works great with my nginx and chfs webdav server! thanks for your contribution!
But I've noticed that some games The Witcher 3: Wild Hunt will be incorrectly named The Witcher 3%3A Wild Hunt, which may be a small urlencode bug.

reviewing the logs and code, urlencode is working fine. The problem here lies in Windows not allowing ":" to appear in file names. And this is automatically handled by my webdav mounting software with %3A, which made me think there was a bug, while the JKSV and Linux webdav server works properly.
I wonder if those who use Windows as a webdav server will encounter similar naming format issues.

[fennectech]

This is due to escaping characters that can be detected as operators in the command or URL used in the background to obtain the content.

[impeeza]

Hello there, I am Using IIS as WebDav and having the same problem, "for the prince of the persia: The Lost Crown"

the WebDav Log on the console have:

WebDav: Create directory at http://192.168.100.8/Entrada/JKSV/Prince%20of%20Persia%3A%20The%20Lost%20Crown/

The IIS Log on my computer have:

2024-02-04 17:32:48 192.168.100.8 PROPFIND /Entrada/JKSV/Prince+of+Persia:+The+Lost+Crown/Impeeza+-+2024.02.04+@+11.53.08.zip - 80 impeeza 192.168.100.17 JKSV - 400 0 0 1
2024-02-04 17:32:48 192.168.100.8 PUT /Entrada/JKSV/Prince+of+Persia:+The+Lost+Crown/Impeeza+-+2024.02.04+@+11.53.08.zip - 80 impeeza 192.168.100.17 JKSV - 400 0 0 1
2024-02-04 17:32:48 192.168.100.8 PROPFIND /Entrada/JKSV/Prince+of+Persia:+The+Lost+Crown/ - 80 impeeza 192.168.100.17 JKSV - 400 0 0 1

On IIS is a pain to handle a URL with ":"

I have another Apache server and also try to create the path:

192.168.100.17 - - [04/Feb/2024:12:44:49 -0500] "PROPFIND /Entrada/JKSV/Prince%20of%20Persia%3A%20The%20Lost%20Crown/Impeeza%20-%202024.02.04%20%40%2011.53.08.zip HTTP/1.1" 405 225
192.168.100.17 - - [04/Feb/2024:12:44:49 -0500] "PUT /Entrada/JKSV/Prince%20of%20Persia%3A%20The%20Lost%20Crown/Impeeza%20-%202024.02.04%20%40%2011.53.08.zip HTTP/1.1" 405 220
192.168.100.17 - - [04/Feb/2024:12:44:49 -0500] "PROPFIND /Entrada/JKSV/Prince%20of%20Persia%3A%20The%20Lost%20Crown/ HTTP/1.1" 405 225

I have changed the game save folder on JKSV SD:/config/JKSV/titleDefs.txt with: 0x0100210019428000 = "Prince of Persia_The Lost Crown" but that is not take in account by WebDav client, could be a good addition, WebDav client to read the folder redirection, What do you think @rado0x54 ?

[rado0x54]

I'm happy to improve on this and other related features in my free time. However first I would really like to have a possibility to get this PR merged as-is. (@J-D-K I know you are busy, but maybe you can just give a quick indication if you would be willing to merge the webdav support in general). Specifically if you are ok to put the Google Drive and Webdav functionality behind the same interface. (e.g. you can have the one OR the other).

[J-D-K]

I'll check it out later. Making them use the same class or namespace is what I would prefer to save having to constantly check with if statements before upload.

[J-D-K]

Also just another note for the future, I'm not 100% sure yet how it will be implemented, but if JKSV is going to support multiple ways of uploading saves, I'm thinking going the inheritance route would be the best bet. Having a massive class or namespace with tons of different functions for different websites and methods just sounds like a mess to me, whereas having a base class with virtual methods sounds a bit easier and cleaner. Just have to figure out how I would like to implement it.

[impeeza]

So far, on my Windows 11 MSYS setup cloned the df89d8b commit, it build fine and is running fine on my console, access to IIS Webdav and Apache (no showing names but is a know bug about how Apache handle the WebDav protocol)

[rado0x54]

Cool @J-D-K thanks for merging. I'm happy to look at your feedback (e.g. inheritance) and incorporate that in a future PR!

[J-D-K]

@rado0x54 I'm currently rewriting the whole project. Once I have it completely functional I'll push it and I'm sure you'll understand how I think it should be approached. I'd branch and push it now, but I don't want people cloning and expecting JKSV as they know it now and having complaints things don't work or are missing.

[impeeza]

@J-D-K , Thanks a lot for all your great work.

[320x200]

@rado0x54 thanks for this feature! It makes a huge difference for managing save files with JKSV!

@impeeza mentioned this discussion in #227 and I have to report the same issue of missing remote save file names with the native WebDAV module from lighttpd. Here is my curl output:

  HTTP/1.1 207 Multi-status
  Content-Location: /path/to/JKSV/
  Content-Type: application/xml;charset=utf-8
  Content-Length: 2304
  Date: Mon, 11 Mar 2024 10:24:54 GMT
  Server: lighttpd/1.4.74

  <?xml version="1.0" encoding="utf-8"?>
  <D:multistatus xmlns:D="DAV:" xmlns:ns0="urn:uuid:c2f41010-65b3-11d1-a29f-00aa00c14882/">
  <D:response>
  <D:href>/path/to/JKSV/</D:href>
  <D:propstat>
  <D:prop>
  <D:getcontentlength>4096</D:getcontentlength><D:getcontenttype>httpd/unix-directory</D:getcontenttype><D:getetag>"677398372"</D:getetag><D:getlastmodified ns0:dt="dateTime.rfc1123">Sun, 10 Mar 2024 19:23:04 GMT</D:getlastmodified><D:res  ourcetype><D:collection/></D:resourcetype><D:supportedlock><D:lockentry><D:lockscope><D:exclusive/></D:lockscope><D:locktype><D:write/></D:locktype></D:lockentry><D:lockentry><D:lockscope><D:shared/></D:lockscope><D:locktype><D:write/><  /D:locktype></D:lockentry></D:supportedlock><D:lockdiscovery></D:lockdiscovery></D:prop>
  <D:status>HTTP/1.1 200 OK</D:status>
  </D:propstat>
  </D:response>
  <D:response>
  <D:href>/path/to/JKSV/Title%20One/</D:href>
  <D:propstat>
  <D:prop>
  <D:getcontentlength>4096</D:getcontentlength><D:getcontenttype>httpd/unix-directory</D:getcontenttype><D:getetag>"2050066314"</D:getetag><D:getlastmodified ns0:dt="dateTime.rfc1123">Sun, 10 Mar 2024 19:23:12 GMT</D:getlastmodified><D:re  sourcetype><D:collection/></D:resourcetype><D:supportedlock><D:lockentry><D:lockscope><D:exclusive/></D:lockscope><D:locktype><D:write/></D:locktype></D:lockentry><D:lockentry><D:lockscope><D:shared/></D:lockscope><D:locktype><D:write/>  </D:locktype></D:lockentry></D:supportedlock><D:lockdiscovery></D:lockdiscovery></D:prop>
  <D:status>HTTP/1.1 200 OK</D:status>
  </D:propstat>
  </D:response>
  <D:response>
  <D:href>/path/to/JKSV/Title%20Two/</D:href>
  <D:propstat>
  <D:prop>
  <D:getcontentlength>4096</D:getcontentlength><D:getcontenttype>httpd/unix-directory</D:getcontenttype><D:getetag>"1780659571"</D:getetag><D:getlastmodified ns0:dt="dateTime.rfc1123">Sun, 10 Mar 2024 23:14:15 GMT</D:getlastmodified><D:re  sourcetype><D:collection/></D:resourcetype><D:supportedlock><D:lockentry><D:lockscope><D:exclusive/></D:lockscope><D:locktype><D:write/></D:locktype></D:lockentry><D:lockentry><D:lockscope><D:shared/></D:lockscope><D:locktype><D:write/>  </D:locktype></D:lockentry></D:supportedlock><D:lockdiscovery></D:lockdiscovery></D:prop>
  <D:status>HTTP/1.1 200 OK</D:status>
  </D:propstat>
  </D:response>
  </D:multistatus>

It looks like displayname is missing indeed...

[320x200]

@rado0x54 I asked on the lighttpd forum about displayname missing. Have a look at https://redmine.lighttpd.net/boards/2/topics/11473 I think it contains some useful info about how the optional property should work, but also why it may be a better idea to find another way to display the name.

[UnderPsi]

Would anyone be so kind as to share a release of the latest build ? I would like to use webdav to easily share saves across multiple devices.

[impeeza]

I don't know if @J-D-K will like, the code is being improved and YOU WILL HAVE NO SUPPORT for that release.

Here is, but please @J-D-K let me know if I have to remove it 2024_02_06-df89d8b-PR207incluido-ESV3.zip

[J-D-K]

@impeeza It's fine. Don't worry about it. But, yeah, I won't be much help with it. Part of the reason I started rewriting everything was to make issues easier to track and fix. Doing it with the master branch is kinda a nightmare now.

[UnderPsi]

@impeeza thank you very much for sharing ! I got it set up on my Synology NAS, managed to upload some saves but as others already observed they show up as [R] with no name and I am not able to download / restore them afterwards. I'll have to wait some more ! Thanks for the excellent work