search

iterate-ch / cyberduck

Cyberduck is a libre FTP, SFTP, WebDAV, Amazon S3, Backblaze B2, Microsoft Azure & OneDrive and OpenStack Swift file transfer client for Mac and Windows.
https://cyberduck.io/
GNU General Public License v3.0
3.39k stars 296 forks source link

Support Manta Object Store #9711

Closed cyberduck closed 7 years ago

cyberduck commented 8 years ago

f6042bc created the issue

Manta is an open source object store like S3 or Swift. It is widely in use and will be quite ubiquitous now that Samsung has purchased Joyent (the creators of Manta).

This ticket is to request that support for Manta be added to CyberDuck.

There is a Java SDK available for Manta.

Attachments

cyberduck commented 8 years ago

@dkocher commented

This is certainly possible but we would need some more users to vote this up to justify the effort.

cyberduck commented 7 years ago

3d620ce commented

Hey there, Tomas from Joyent here, I've been using Cyberduck for years now and want to contribute to the project. I'm researching what it would take to implement this and had a look at ticket #9799 along with the associated changeset which added support for Microsoft OneDrive. After reviewing the changes and getting project building locally, I'm interested in implementing a storage provider for Manta within Cyberduck. Following the example of 71fd632b3ec145addc754042b951bd6dc10fe295, it looks like the steps for adding a new provider is as follows:

Please let me know if I've missed any critical details or there are any points in need of clarfication. The divergence in terminology for the authentication secret stems from the fact that authentication for Manta relies on a combination of username and an SSH Key fingerprint for a key that has previously been uploaded to the Manta instance in use (see Setting Up Your Environment in the Manta documentation for details). Since Cyberduck provides a dropdown field for supplying an SSH Private Key in addition to text input for passwords there is some ambiguity regarding the best approach to take, please see Question 1 for some proposals about how to handle this.

Questions:

  1. The Open Connection dialog includes fields for both Password/AuthCode and an SSH private key. This gives us some options for authenticating users (since the labels are configurable, Password will refer to the text input and SSH Private Key refers to the dropdown that is enabled when the SFTP protocol is chosen):
    1. disabling the Password field and expecting users to select a SSH Private Key. This seems to be the most intuitive to handle key selection.
    2. disabling the SSH Private Key field and expecting users to supply the key fingerprint in the Password field, similarly to the way the Manta documentation asks users to set the MANTA_KEY_ID environment variable. This aligns with Manta CLI configuration but is not user-friendly.
  2. Sonar is mentioned on the main page of the issue tracker but is inaccessible through https://sonar.cyberduck.io/ and I couldn't find any style-checking plugin in the project, is there a different code style tool in use?
  3. The modifications in https://trac.cyberduck.io/changeset/38915/trunk/Cyberduck.xcodeproj/project.pbxproj refer to resources being added to the Xcode Project. Are there any special considerations around adding to this file or would I be able to make this change myself?
  4. The java-manta library allows for optional Client-Side Encryption while the Cyberduck roadmap indicates support for Cryptomator is being added. Handling encryption through Manta's client library could lead to issues and confusion, in addition to requiring us to either collect more configuration options or force certain settings from Cyberduck. How the Manta provider in Cyberduck will support encryption is still an open question I'd like to discuss further once these questions are ironed out.
cyberduck commented 7 years ago

@dkocher commented

Fixed the sonar link to point to (https://sonar.iterate.ch/).

cyberduck commented 7 years ago

@dkocher commented

The codestyle is part of the project in the .idea directory and picked up automatically when opening the project. Obviously this only works when working with IntelliJ IDEA which is our preferred IDE.

cyberduck commented 7 years ago

@dkocher commented

I would suggest to move the client-side encryption support in Manta to a different ticket. Our Cryptomator integration is now finished and will be part of 6.0 available later today.

cyberduck commented 7 years ago

@dkocher commented

You will need XCode installed to make changes to the project.pbxproj project file. Adding resources there is not a priority and can be done at the very last to include resources by default. You can install your new connection profile by double clicking – see Profiles.

cyberduck commented 7 years ago

@dkocher commented

For the ambiguity for the SSH key fingerprint required you will want to at least override getPasswordPlaceholder from the Protocol interface. This will adjust the label and placeholder text when editing the bookmark. Or you can disable the password field with isPasswordConfigurable. Currently the private key popup is only enabled for protocols of type sftp (see getType in Protocol).

cyberduck commented 7 years ago

@dkocher commented

You analysis about what is to implement is right. Don't hesitate to ping me for any further clarifications.

cyberduck commented 7 years ago

3d620ce commented

Work on this is progressing though I've got two questions thus far:

When attempting to build and run the project in Xcode (using Cyberduck.xcodeproj to open the project) the application crashes immediately with the error pasted below. I haven't had a chance to debug it further but noticed there was no output in the Xcode console and am not familiar with logging in Objective-C. Any suggestions?

Error loading runtime bundle.
The application failed to launch with a unrecoverable exception.

When attempting to turn on debugging using the debug.sh script I get an error about paths, though there is a file at that path it's missing the .debug suffix:

  [cyberduck](master) PDT 17:13 UTC 00:13
./debug.sh -e
Enabling debug configuration...
/Users/tomascelaya/repo/cyberduck
cp: /Users/tomascelaya/repo/cyberduck/setup/app/Info.plist.debug: No such file or directory

Not sure how to resolve the first issue further without getting past the second.

cyberduck commented 7 years ago

@dkocher commented

Create a debug configuration Info.plist.debug yourself by duplicating the Info.plist file and prepend -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 to the value for VMOptions. I will attach a sample.

cyberduck commented 7 years ago

@dkocher commented

I never run the project from XCode – the build in XCode is partial and only covers the native libraries. You will have to run mvn package -DskipTests -pl osx -am.

You can then execute the Cyberduck.app binary in osx/target or run with the i18n.sh script using i18n/src/main/resources/i18n.sh -l en.lproj -r.

cyberduck commented 7 years ago

@dkocher commented

We are now mirroring to https://github.com/iterate-ch/cyberduck to make contributions easier.

cyberduck commented 7 years ago

3d620ce commented

Thanks for all the help so far! Definitely getting closer sincemvn package succeeds without any errors. Running the bundle at osx/target/Cyberduck.app closes as soon as it's launched though and invoking open osx/target/Cyberduck.app yields the following error:

LSOpenURLsWithRole() failed with error -10810 for the file /Users/tomascelaya/repo/cyberduck/osx/target/Cyberduck.app.

Thankfully i18n.sh yield a more obvious error message:

  [cyberduck](master) PDT 14:39 UTC 21:39
i18n/src/main/resources/i18n.sh -l en.lproj -r
Using Language:en.lproj
Running app using en...
Error: Could not find or load main class ch.cyberduck.ui.cocoa.MainApplication
*** DONE. ***

The error specifically occurs on the last line in run() but attempting to run the Cyberduck.app/Contents/Cyberduck binary prints out the same message, so it seems like I'm missing something related to classpath configuration. I'm attempting to track down the issue further by inspecting file open syscalls but haven't found anything promising yet.

cyberduck commented 7 years ago

@dkocher commented

Please try running the install phase with mvn install -DskipTests -pl osx -am.

cyberduck commented 7 years ago

3d620ce commented

That did the trick! A warning came up about "App Transport Security" but everything seems to work and the manta endpoints are all https so I don't think it'll cause any issues.

Thanks!

Replying to [comment:17 dkocher]:

Please try running the install phase with mvn install -DskipTests -pl osx -am.

cyberduck commented 7 years ago

3d620ce commented

I'm nearing completion and will be ready to submit my changes as soon a I resolve an issue with Manta subusers, would a pull request on Github work or is there some other process for submitting contributions?

cyberduck commented 7 years ago

@dkocher commented

Replying to [comment:19 tjcelaya]:

I'm nearing completion and will be ready to submit my changes as soon a I resolve an issue with Manta subusers, would a pull request on Github work or is there some other process for submitting contributions?

A pull request on GitHub works just great.

cyberduck commented 7 years ago

@dkocher commented

Now in feature/TRAC-9711.

cyberduck commented 7 years ago

@dkocher commented

Pending upstream issues

cyberduck commented 7 years ago

3d620ce commented

java-manta 3.7.1 3.1.7 (or 3.8.0 3.2.0) is undergoing performance testing and will be released very soon. In the meantime I wanted to try building Cyberduck on Windows and have been running into an issue with Maven/Ant/MSBuild. I've installed the following software:

cyberduck commented 7 years ago

@dkocher commented

Please try the branch (https://github.com/iterate-ch/cyberduck/tree/feature/TRAC-9711) and install the build tasks from (https://github.com/loresoft/msbuildtasks).

cyberduck commented 7 years ago

3d620ce commented

Thanks, I installed the latest msbuildtasks MSI and am getting further into the build but have encountered some confusing errors. I've attached them as windows-build-failure-2017-09-26.log.

Which version of the .NET Framework and Windows SDK is used for builds? I discovered that I needed to install the Windows 8 SDK (https://developer.microsoft.com/en-us/windows/downloads/windows-8-sdk) based on the error message that occurred after adding msbuildtasks but the new errors seem to be related to compiler versions? In order to install as few conflicting version of .NET as possible I am using the MSBuild executable from the .NET Framework installed by Windows 8 SDK ( added C:\Windows\Microsoft.NET\Framework\v4.0.30319 to my PATH since it contains MSBuild.exe). These errors are the same on both Windows Server 2012 and Windows Server 2016.

cyberduck commented 7 years ago

@AliveDevil commented

Using MSBuild from C:\Windows\Microsoft.NET\Framework\v4.0.30319 will not work as we are using features not present in this version.

Please install Visual Studio 2017 from https://www.visualstudio.com/en/vs/ and include following "additional items" upon installation. "Workloads" required to build Cyberduck for Windows:

Additional items required to build Cyberduck for Windows or Cyberduck for Windows Store:

For more information about workloads and additional items in visual studios new installer go to https://docs.microsoft.com/en-us/visualstudio/install/modify-visual-studio

You will, however, get additional errors after building as we create an installer just after building. You can ignore those errors.

cyberduck commented 7 years ago

3d620ce commented

The Visual Studio 2017 installation (w/ Windows 10 SDK (10.0.14393.0)) seems to have addressed some compiler issues. Unfortunately I've encountered a new error. Apologies for stumbling on what are most likely very simple issues, I really appreciate the assistance!

The latest errors have been attached as windows-build-failure-2017-09-27.log.

cyberduck commented 7 years ago

@dkocher commented

Replying to [comment:28 tjcelaya]:

The Visual Studio 2017 installation (w/ Windows 10 SDK (10.0.14393.0)) seems to have addressed some compiler issues. Unfortunately I've encountered a new error. Apologies for stumbling on what are most likely very simple issues, I really appreciate the assistance!

The latest errors have been attached as {{windows-build-failure-2017-09-27.log}}}.

We have updated the build instructions at (https://github.com/iterate-ch/cyberduck/blob/master/README.md). Please install Bonjour SDK for Windows or exclude the bonjourmodule from the build.

cyberduck commented 7 years ago

3d620ce commented

Thanks for all the help! It looks like I'm down to one last issue, actually loading the new cyberduckprofile... I've added a new MantaProtocol() with the appropriate using statement at the top of MainController.cs but the profile doesn't actually show up when invoked. Confirmed profiles could be added using Microsoft OneDrive Business.cyberduckprofile. Other steps taken include adding the new profile into profiles/default, including the <Bundled/> property set to true, including an image with the protocol name in ResourcesBundle.resx using Visual Studio 2017 on the top-level .sln project. I know the issue is related to loading the protocol because repurposing a different profile which is confirmed to work and changing its protocol leads to the same behavior.

In search of a commonality between the protocols which did load I tried grepping for protocol names in groups (i.e. grepping individually for each protocol) and combining the lists with comm -12 but alas the only results were the git index and compiled files like .dll, .exe. It would help if I could figure out how to see the logs printed to stdout but I haven't figured out how to do that (or if it's possible) yet.

Basically, how do I make sure the Windows build can use the new protocol?

cyberduck commented 7 years ago

@dkocher commented

Please move the connection profile to profiles/default and install a snapshot build of the profiles module prior building if you want to have the profile bundled by default.

I have fixed the missing dependency in the feature brach.

cyberduck commented 7 years ago

3d620ce commented

Thanks again for all the assistance! I'm doing a final pass of acceptance testing now that Windows builds are picking up our cyberduckprofile and will be submitting one more PR against feature/TRAC-9711 that should make the manta protocol implementation ready for release.

cyberduck commented 7 years ago

@dkocher commented

In d52c5d818c29c32ef3dcf86288c4386755b213a8.

cyberduck commented 7 years ago

3d620ce commented

I'm not sure why this behavior wasn't occurring on the feature/TRAC-9711 branch but it seems like the password entry field is now disabled when trying to log in with a passphrase-protected key. Still bisecting the commits between master and the previous working version to figure out where the bug was introduced so I can provide more detail about the issue.

cyberduck commented 7 years ago

@dkocher commented

Replying to [comment:35 tjcelaya]:

I'm not sure why this behavior wasn't occurring on the feature/TRAC-9711 branch but it seems like the password entry field is now disabled when trying to log in with a passphrase-protected key. Still bisecting the commits between master and the previous working version to figure out where the bug was introduced so I can provide more detail about the issue. I will have a look. Possibly a regression that also applies to SFTP.

cyberduck commented 7 years ago

@dkocher commented

Replying to [comment:35 tjcelaya]:

I'm not sure why this behavior wasn't occurring on the feature/TRAC-9711 branch but it seems like the password entry field is now disabled when trying to log in with a passphrase-protected key. Still bisecting the commits between master and the previous working version to figure out where the bug was introduced so I can provide more detail about the issue. In 4f8182b7aeaf899d0cbc411438a0f17c352253d8.

cyberduck commented 7 years ago

3d620ce commented

Thanks for the quick resolution. I opened a PR to address the fact that the private key file dropdown is enabled and might be confusing (plus a URL encoding issue)

cyberduck commented 7 years ago

@dkocher commented

Replying to [comment:38 tjcelaya]:

Thanks for the quick resolution. I opened a PR to address the fact that the private key file dropdown is enabled and might be confusing (plus a URL encoding issue)

Changed to password prompt for private key passphrase in 6a15da9501963b710765308eec55c36196c918fe.