bernd-wechner / Degoo

CLI tool(s) for working with Degoo cloud storage
Other
143 stars 41 forks source link

FUSE? #38

Open rcfa opened 2 years ago

rcfa commented 2 years ago

Thanks! This looks great, as I theoretically 13TB lifetime with them, which I can't use because they changed/abandoned their client software. (Don't get, why they don't at least expose a WebDAV interface to their storage.)

That said, 13TB, even with CLI tools aren't usable for much without lots of wasted time uploading, checking, etc.

Ideal would be a FUSE interface ( https://wiki.osdev.org/FUSE ), so the storage can be mounted as a network drive, and then used with random backup, file-sync, etc. software, without degoo specific code. Is that a direction you plan to take your code in?

The reason I ask: you seem to have already implemented the type of operations that sftp offers, and there's a FUSE abstraction layer based on those ( https://github.com/libfuse/sshfs )

rcfa commented 2 years ago

Actually, there's another way this project's effort could be used. There's the rClone https://rclone.org https://github.com/rclone/rclone cloud sync effort, which also allows for FUSE mounting of cloud storage.

They don't support degoo however, for lack of an API.

Maybe, with your work and their work combined, rClone could bring support for degoo...

bernd-wechner commented 2 years ago

You took the words right out of my mouth, actually. I have been meaning to drop rclone a line to see if they will take the reverse engineering I have captured here in order to add the Degoo service to the list of services they support. As I have many distracting priorities at present if you want to follow that up please do. We can only hope that some rclone contributor(s) might be excited by the opportunity to add another provider to their list with the API, while unspecified, deconstructed at least in part here:

https://github.com/bernd-wechner/Degoo/blob/master/degoo/API.py

rcfa commented 2 years ago

https://github.com/rclone/rclone/issues/4424

There's this thread. They seem to be hesitant about reverse engineering APIs, but non-Windows platforms support supposedly pluggable backends through shared libraries. So you could create one of these, and at least all the *IX platforms would be able to use it. And if things are reasonably stable, they might absorb it. Or they might do it from the start, if someone else is maintaining the relevant code and they don't have to worry about it.

rcfa commented 2 years ago

Just also found this, which seems to build on your work:

https://github.com/MDKPredator/degoo_drive

forrie commented 2 years ago

Re: rclone

I use this from time to time, and I can't believe I didn't think of suggesting this LOL I think it would be a perfect fit for rclone's model.

Regarding the reverse engineering of protocol, if you have a relationship with someone at Degoo, perhaps you can include that mention -- or even ask your contact at Degoo how they feel about it. That statement could be more convincing. But if Degoo isn't cooperating, that might be viewed differently by the rclone folks?

In any case, great idea rfca :)

bernd-wechner commented 2 years ago

Alas, no contact at Degoo. I have had email exchanges with support but they have declined to offer any technical information on their API. They are aware of this reverse engineering.

forrie commented 2 years ago

I have been utterly puzzled by Degoo's apparent apathy. I bought the 10TB "forever" deal, from Cult of Mac, a long time ago and have had a lot of frustration with them. I understand they may be a small shop; however, it's really in their best interests to make the platform more accessible, usable, which this project being important in context.

If the rclone folks agree to incorporate interfacing code, then someone is going to need to keep on it and analyze the changes Degoo makes, updating the rclone code accordingly. Seems like a daunting, frustrating task... especially if Degoo couldn't care less. shrug

sskras commented 2 years ago

@bernd-wechner commented on Feb 21:

We can only hope that some rclone contributor(s) might be excited by the opportunity to add another provider to their list with the API, while unspecified, deconstructed at least in part here:

https://github.com/bernd-wechner/Degoo/blob/master/degoo/API.py

@rcfa commented on Feb 21:

non-Windows platforms support supposedly pluggable backends through shared libraries. So you could create one of these, and at least all the *IX platforms would be able to use it. And if things are reasonably stable, they might absorb it.

@forrie commented on Feb 21:

If the rclone folks agree to incorporate interfacing code, then someone is going to need to keep on it and analyze the changes Degoo makes, updating the rclone code accordingly. Seems like a daunting, frustrating task...

Folks, I would summarize you the following way:

it would be nice if someone implements this as a standalone, dedicated lib, eg. libdegoo, and then optionally maintains it – continuously tests for breakages in the De-facto API and reimplements the changes back into the lib.

So this needs like at least one dedicated person who:

... right?

ItachiSan commented 2 years ago

Hi all, I am quite interested in this chat. I found this project (was thinking of reverse engineering Degoo myself for a small Dropbox-like app). The reason I didn't start doing it is because of the license; while I wholesomely agree, I am unsure if it is copyleft or not.

In short, my idea is:

I thus want to verify the following whether the license is GPLv3 like and thus everything is should be under this license. The reason is that, if I reimplement stuff, I will properly give credits and such, but having the responsibility to verify contributors to e.g. not having criminal records that are against human rights sounds hard as an individual (I am considering the extremes of the license here)

bernd-wechner commented 2 years ago

@sskras, nice summary. I cover some of those qualities but alas not the first one: "dedicated" ;-) . Mainly as I'm stretched across far too many commitments. It seems reverse engineering the API motivated me a while ago but I've got distracted and stuck on the login issue (unable thus far to solve that in Python as we now understand its causes just not a Python solution). Yet. Once that's solved, woo hoo. Of course yes, then a basic test suite would be a priority so we can keep up to date on status of the API modelling.

But ...

@ItachiSan, you are right all round, I think a FUSE interface and a sync client would both rock and help us get value out of our TB of Degoo storage that Degoo are selling (still) but have no useful interface too enabling us to do backups to it. And why not Rust, go for it, anyone with time and enthusiasm to work on a FUSE interface and get syncing working better has my blessings (one of the sync issues I've had is that Degoo don't provide server side file hashes and so change detection is not slick).

I'm not sure what you mean by license. The Degoo API has no license that I'm aware of it's a publicly exposed API, their website uses it and exposes it, and I see no licensing notes anywhere, having assumed that if you have an account and can log in, you are licensed to use their storage services. It would surprise me if they were so obstructionist as to encode in a license some restriction to using clients they write, least of all when they won't write a decent one.

As to my reverse engineering the license is here:

https://github.com/bernd-wechner/Degoo/blob/master/LICENSE.md

more liberal than the GPL by a ways ;-) and yes, mainly concerned with ethical application ....

P.S. Your username conjures unshakeable images of Japanese in my mind (Itachi san ...) and yet you're apprently Swedish with an Italian name ... love the diversity (trumps mine even ;-).

vinibali commented 1 year ago

Hi! I'm here to push you guys towards to talk about rclone again. Degoo's plan is super impressive, we could benefit a lot if we could use rclone for this provider. What do you think?

bernd-wechner commented 8 months ago

I found this in the interim:

https://thepythoncorner.com/posts/2017-02-27-writing-a-fuse-filesystem-in-python/

And I got login working again. But have yet to finish testing. Have my fingers it way too many pies. This is slow and on my backburner but with login fixed I found the API had progressed and so had to update some of the API interface and am not done with that. Tested all utils bar put and get (the important ones ;-).

Looks like pyfuse can help us, though I do wonder about how laggy and sluggish a fuse mounted drive will be ;-). Will implement it some time and see.