Make v1.3 work with Mono

[luckyrat]

I've had lots of trouble getting the new version to work with Mono. However, I probably won't have access to my development machine for a couple of weeks so I'm hoping that @dlech or @krbvroc1 or others might have some advice to offer in the mean time.

So far I've been trying with Mono 2.10.8.1 on Ubuntu 13.04 (KeePass 2.23 official zip version) but hit a number of problems.

1) The plgx file does not load. --saveplgxcr outputs a list of warning messages but no errors. One suspicious warning message is about "The predefined type 'System.Runtime.CompilerServices.ExtensionAttribute' is defined in multiple assemblies" which is due to the way that the .NET 2.0 version of the Fleck web sockets library has worked around the lack of support for extensions in .NET 2.0 (https://github.com/peters/Fleck2/blob/master/src/FleckExtensions.cs#L9). Maybe that is at least part of the reason that it won't compile.

2) When compiling KeePassRPC directly on the Linux box (using @dlech's useful guide: https://discussions.zoho.com/keefox/topic/keefox-1-2-7-keepassrpc-plgx-won-t-complile-with-mono-3-0-6#71684000000346007) it compiles OK and when I move the compiled and referenced DLLs to the KeePass plugin folder and start KeePass, it loads up without an error. However, it does output an exception to the console which may be a sign of a problem (a Fleck2.WebSocketException).

3) The initial stages of the security negotiation work but then the "Authorise a new connection" dialog appears with completely black contents. I guess I'm doing something that Mono doesn't support yet but have a small hope that it's a more isolated problem with the specific version of Mono that ships with Ubuntu or the graphics drivers for my VirtualBox VM.

All of this means that I'm still not sure whether the System.Security.Cryptography.ProtectedData.Protect() function I mentioned in #194 will be a problem in Mono but one step at a time!

Any ideas where I should look next or any insight you can provide from trying the build on your Mono-based machines?

[krbvroc1]

It has been a long time, but I will setup my development environment and see what I can do to help out, troubleshoot, and hopefully get it working. Will you be avail for questions if I run into some?

[dlech]

Re: 1)

One suspicious warning message is about "The predefined type "System.Runtime.CompilerServices.ExtensionAttribute' is defined in multiple assemblies"

Yup. That is the one.

.NET 2.0 version of the Fleck web sockets library has worked around the lack of support for extensions in .NET 2.0

Very clever, but not good for one binary that will run everywhere. Have you considered dropping support for .NET 2.0? Really it only matters to Windows XP users and with support for XP ending April 2014, there are hopefully not many left (an 3.5 is probably good enough for most holdouts too). The other option would be to include multiple versions of the Fleck dll in the plgx and use a pre-compile command to copy the correct one.

In the plugin compiling code in KeePass, the 2.0 compiler is never explicitly called. So using the Fleck 2.0 dll is just not going to work on Mono.

[dlech]

In the plugin compiling code in KeePass, the 2.0 compiler is never explicitly called. So using the Fleck 2.0 dll is just not going to work on Mono.

Just thought of this: you could change the namespace in the Fleck library so that it does not conflict.

[luckyrat]

Yeah I'll be around for questions but probably not every day.

I have considered dropping .net 2 support but I'd rather not until at least after Microsoft drop XP support and preferably a few months or year later to give people more time to upgrade. Since the problem only occurs in .net 2, we'd probably only need two versions of the DLL but renaming the namespace sounds neater if it works. Although I wonder if we would then hit problems with ambiguous references in everything except .Net2.

[dlech]

I am having issues getting the Firefox Addon portion of the 1.3dev branch to work. Is it in a usable state? I am getting lots of errors about cannot find keefox_org in KF.js. I though this was one of the files that was supposed to be a .jsm.

[luckyrat]

I refactored those files so they no longer need to be combined (now using module imports in a better way). I think this information can be seen in the diff of the csproj but maybe it's in a file that doesn't get committed to git.

The fams files still need to be combined but I'll try to change that too soon.

It should be in a usable state but let me know if you have any more trouble. On 27 Oct 2013 21:20, "dlech" notifications@github.com wrote:

I am having issues getting the Firefox Addon portion of the 1.3dev branch to work. Is it in a usable state? I am getting lots of errors about cannot find keefox_org in KF.js. I though this was one of the files that was supposed to be a .jsm.

— Reply to this email directly or view it on GitHubhttps://github.com/luckyrat/KeeFox/issues/212#issuecomment-27179644 .

[krbvroc1]

Having trouble getting a development environment setup.

@dlech How are you building the plugin under Linux? I tried using xbuild and was trying to use the resulting DLL file but KeePass doesn't even load it. It seems to me that when building the DLL, the 'File Version Information Block' that KeePass examines to determine it is a plugin is empty. On windows, you right click the DLL and can get/set that info. Not sure under Linux/Mono.

I'm the past I think I had to compile the plugin under windows and copy it over to Linux - a time consuming process that I thought may have been solved by using xbuild.

Any suggestions?

[dlech]

It sounds like you have Mono 3.0.x. There is a known bug that prevents the assembly info from being generated and as you discovered, KeePass will ignore the dll if it does not have the expected info.

The command on linux to check is monodis --assembly KeePassRPC.dll.

I'm actually building a .plgx on linux using a tool I wrote (mostly copied from KeePass).

# assuming you are on ubuntu
sudo apt-add-repository ppa:dlech/keepass2-plugin-dev
sudo apt-get update
sudo apt-get install keepass2-plugin-dev
plgx-tool --build --input=KeePassRPC --output=KeePassRPC.plgx

As for the Firefox part of the plugin, I am just using an extension proxy file to link to the Firefox Addon\KeeFox directory. As mentioned above, you have to combine FAMS.js and FAMS-config.js into FAMS.jsm (cat FAMS.js > FAMS.jsm; cat FAMS-config.js >> FAMS.jsm). There appears to be something else I am missing though as there are errors and the FireFox extension does not work properly. I haven't figured out what yet though. Also, it is for FireFox 25+ only.

[krbvroc1]

I am actually using Fedora 19 which uses mono 2.10.8. This has been a known good configuration for me and I wanted to try to get things working before messing with any other mono versions.

I also tried openSUSE 12.3 since it contains a never mono environment (3.0.4) which is probably where I saw the missing version info.

I'm trying to run the DLL so I can have access to debugging and debugging symbols. On 2.10.8, I run into 'Could not load type 'KeePassRPC.KeePassRPCExt' from assembly 'KeePassRPC, version=2.0.15.32792, Culture=neutral, PublicKeyToken=null'.

Basically, in PluginManager, the Activator.CreatInstanceFrom() fails on the DLL. Not sure if DLL loading is just not possible or what.

That monodis command prints pretty much the same on 2.10.8 and 3.0.4:

Assembly Table
Name:          KeePass
Hash Algoritm: 0x00008004
Version:       2.23.0.25849
Flags:         0x00000000
PublicKey:     BlobPtr (0x00000000)
    Zero sized public key
Culture:       

[krbvroc1]

UPDATE: It seems that when running the DLL, I needed to also copy the Fleck2, Jayrock, Mono.Security, DLLs into the plugin directory.

[krbvroc1]

After creating the extension proxy file (I think I used a symbolic link a two years ago when I worked on this), I got the popup to 'Authorise a new connection' and an SRP code.

After entering that, I was able to create a new database. Not sure about the FAMS issue you mentioned. I just seem to be stuck on 'KeeFox loading...' on the browser, but haven't looked into that yet. I am seeing

--
[20:31:53.371] Mon Oct 28 2013 20:31:53 GMT-0400 (EDT):DEBUG: Refreshing KeeFox's view of the KeePass database.
[20:31:53.371] Mon Oct 28 2013 20:31:53 GMT-0400 (EDT):INFO:  At least one database is open.
[20:31:53.371] Mon Oct 28 2013 20:31:53 GMT-0400 (EDT):WARN:  Could not refresh KeeFox in a window. Maybe it is not correctly set-up yet? TypeError: this._currentWindow is null
[20:31:53.371] Mon Oct 28 2013 20:31:53 GMT-0400 (EDT):INFO:  KeeFox feels very refreshed now.
[20:33:50.072] Mon Oct 28 2013 20:33:50 GMT-0400 (EDT):DEBUG: fillCurrentDocument start
[20:33:50.072] TypeError: this._currentWindow is null @ chrome://keefox/content/KFToolBar.js:854

[dlech]

I found the error (issue #214). I am guessing that Ken was using an existing profile that already had the mono binary location set. I started with a fresh profile and KeeFox would not start properly. Now, back to the plgx issues...

[dlech]

By the way, the authorization screen looks good to me using Mint 15 (=raring) in a VirtualBox VM. Ubuntu hasn't updated mono in ages, so I am not sure what the issue is on Chris's install.

[krbvroc1]

@dlech You mentioned something about Firefox 25+ only when you were mentioning FAMS. This plugin doesn't require 25+ does it? Wondering if that is related to my KToolBar error with null currentWindow error?

I found this bug https://bugzilla.mozilla.org/show_bug.cgi?id=875138 related to the devtools (I am seeing error messages in the developer console related to radio buttons). It refers to https://bugzilla.mozilla.org/show_bug.cgi?id=653881.

Looks like Firefox 25 was just released today...Not sure how long until it hits the repository for my Fedora 19 distro.

[dlech]

Chris can explain better, and I am pretty sure that he wants to support Firefox <25 with 1.3. However, when I first tried to run KeeFox in Firefox 24 though, I noticed one of the XUL elements was causing an error that said it did not exist. This leads me to believe that Firefox 25 is needed (or preferred anyway).

[krbvroc1]

@dlech Yep - FAMS missing .jsm was the issue being stuck on the toolbar loading. That got me passed the stuck Keefox loading... Now I am stuck as 'Logged out'.

Clicking Logged does nothing. (however it it communicating with keepassrpc).

If I select the database via KeePass itself, the toolbar changes to logged in. Attempts to use the KeeFox tutorial appears to work.

so it seems using the .DLL on Fedora 19, after creating the fams .jsm files works. Mono is 2.10.8. I get a feeling that perhaps Fedora and Ubuntu (and perhaps other distros) have refused to upgrade Mono to 3.x due to its bugs.

[krbvroc1]

@dlech I am seeing the following:

[11:26:18.953] Tue Oct 29 2013 11:26:18 GMT-0400 (EDT):INFO:  KeePass install location found in preferences: /home/kbass/KEEFOX_1_3_BIN
[11:26:18.953] Tue Oct 29 2013 11:26:18 GMT-0400 (EDT):DEBUG: Looking for the KeePass EXE.
[11:26:18.953] Tue Oct 29 2013 11:26:18 GMT-0400 (EDT):INFO:  KeePass EXE found in correct location.
[11:26:18.953] Tue Oct 29 2013 11:26:18 GMT-0400 (EDT):INFO:  keePassRPC install location found in preferences.
[11:26:18.953] Tue Oct 29 2013 11:26:18 GMT-0400 (EDT):INFO:  Looking for the KeePassRPC plugin plgx
[11:26:18.954] Tue Oct 29 2013 11:26:18 GMT-0400 (EDT):DEBUG: KeePassRPC PLGX search threw an exception: [Exception... "Component returned failure code: 0x80520006 (NS_ERROR_FILE_TARGET_DOES_NOT_EXIST) [nsILocalFile.isFile]"  nsresult: "0x80520006 (NS_ERROR_FILE_TARGET_DOES_NOT_EXIST)"  location: "JS frame :: resource://kfmod/utils.js :: Utils.prototype._confirmKeePassRPCInstallLocation :: line 269"  data: no]
[11:26:18.954] Tue Oct 29 2013 11:26:18 GMT-0400 (EDT):INFO:  KeePassRPC DLL found in correct location.

That is probably what you were referring to? I believe that is because it looks for the .plgx file. Myself, I am using a .dll so that will not be found and the exception is expected. You mentioned you had not set an installation location in your preferences initially, so maybe that is why you encountered that?

[dlech]

I guess I didn't look carefully enough before, but I just tried Firefox 24 again. It looks like the XUL error is from something else - not KeeFox, so never mind the comments about Firefox 25.

[dlech]

3) The initial stages of the security negotiation work but then the "Authorise a new connection" dialog appears with completely black contents. I guess I'm doing something that Mono doesn't support yet but have a small hope that it's a more isolated problem with the specific version of Mono that ships with Ubuntu or the graphics drivers for my VirtualBox VM.

I was able to reproduce this issue. It appears to happen when KeePassRPC starts while KeePass is compiling another plugin that fails. When the compiler tries to show the error message, there is a deadlock with the authorization dialog. We need to find a way to ensure that KeePassRPC does not start or at least does not show any dialogs before KeePass is done compiling plugins.

One way to do this might be to make this async by replacing MainForm.Invoke with MainForm.BeginInvoke.

[luckyrat]

Keepassrpc needs to pause the authentication process at that point so the change would need to be a bit more substantial than begininvoke. I expect it will be possible but since the srp process is normally synchronous we'll have to be careful that we don't introduce any security risks.

I expect we'll need to find a way to make this asynchronous since other plugins or things like keepass update checks might also cause the same problem.

[krbvroc1]

Not sure of the ramifications, but maybe asking KeePass to make a change over there? Maybe the plugin API should have something after Initialize that indicates all plugins have been loaded--like AllLoaded? Or can this happen other times too?

[dlech]

I don't see any events, but you can probably hook into the trigger system here or here

[krbvroc1]

Based on a comment elsewhere, Dominik, mentioned

bool bNoDialog = (KeePass.UI.GlobalWindowManager.WindowCount == 0);

That would tell you there are any sub-windows open under KeePass. So you could delay until then. There was also a mention of using MainForm.UIBlockInteraction() to block any other windows opening which could be used while displaying your window.

Kicking off those checks from one of the triggers mentioned above might be a good technique.

[luckyrat]

I still don't really understand why this code is deadlocking. If my call to Invoke() is processing on the main thread, why doesn't that just continue and block whatever called onto the main thread after my Invoke - sure, it's not a good situation and I'm still expecting to have to make changes to the SRP protocol to cope with it but I'm not sure why it should deadlock. I wonder if Mono is doing something a bit different with regards to graphics rendering off the main thread which somehow causes the main thread to yield in a way that doesn't happen in Windows... or maybe I've just been exceptionally (un)lucky to not see this problem when running on Windows.

I looked into the KeePass plugin system for an "all loaded" event (and other plugin compilation metadata) a couple of years back when investigating localisation of plgx files and couldn't find anything of use. I've not noticed anything in subsequent release notes to suggest that's changed so I'm not surprised you've found nothing too.

However, in this case I think that such an event might not completely solve the problem. Unless we can be sure that the deadlock is specific to something that happens only during application startup, we should assume that the same issue can be triggered by other application or plugin background threads and therefore try to find a way to avoid blocking the main thread by switching to BeginInvoke().

Thanks both for your time looking into things recently, I'm back at my normal dev machine now so hope to make some progress on this at the weekend, if not before.

[dlech]

I wonder if Mono is doing something a bit different with regards to graphics rendering off the main thread which somehow causes the main thread to yield in a way that doesn't happen in Windows...

I think that the issue is that in Windows, you can call System.Windows.Forms.MessageBox.Show () method from any thread and it will work because it uses a Windows API call and the only interaction with the main thread is blocking it until the message box is closed. In fact, one way of looking at is is that on Windows, MessageBox is almost like running a completely separate program. In Mono, I think that MessageBox is treated like any other window and so has to created and displayed from the main thread. As a result, you have to be much more careful with your use of MessageBox. In KeePass, the MessageService uses a message box for some of its messages and this is where the hangup is coming from.

I might mess with it a little more tomorrow. I am wondering if we could make the KeePassRPC authorization dialog modal then maybe that would fix the problem.

[luckyrat]

That explanation makes sense.

The modal workaround sounds promising.

[luckyrat]

Trying to get my head around the warning about duplicate ExtensionAttribute definitions... am I just being overly optimistic to think that this is not actually a problem after all and the failure to compile was more related to the Mono.Security issue? Have either of you got the plgx to compile with the latest code?

If it is still a problem, the only way I can think to resolve it is to do some kind of pre-compilation DLL renaming trick because renaming the namespace won't work since the compiler needs to find the ExtensionAttribute class in the System.Runtime.CompilerServices namespace defined in both Fleck2 and System.Core.

I'm not against bundling a couple of versions of Fleck2.dll if that solves the problem (they're less than 50KB each) but I'm not sure if I need to bother.

[krbvroc1]

@luckyrat I was using the DLL only, not the plgx - with whatever version was current about 9 days ago.

[luckyrat]

I've just tried a new plgx based on the most recent code and it still fails to load via Mono. The strange thing is that I have just tried running the exact build command KeePass says it has used and that works fine (multiple warnings but a successful build; build needs to be run while "compilation failed" messagebox is still displayed to avoid the temp directories being cleared).

So it looks like KeePass is reporting a spurious build failure or the build only works when executed for a 2nd time, I'm not yet sure what could cause either situation...

[dlech]

The underlying problem is probably technically a bug in mono. KeePass uses the C# CodeDom compiler, which returns a CompilerResults object. The compiler errors and warnings are kept in the Errors property. The warning about Fleck2.dll containing a duplicate definition of System.Runtime.CompilerServices.ExtensionsAttribute contains 2 child messages. Mono seems to treat these as errors even though the parent "error" is just a warning.

Anyway, I have sent 2 separate pull requests. Each one will fix the problem, so take your pick. I have tested #216 in a plgx and it compiles and everything seems to work (although not extensively - just did first page of KeeFox tutorial).

[luckyrat]

Definitely sounds like a Mono bug. Thanks for finding some workarounds.

I think I'll go for the changes to the Fleck2 library (hope to have time to pull the fix this evening) mainly because I've already had to make changes to the library to fix another bug and the Mono bug we're working around here is not specific to KeePassRPC (or even KeePass plugins in general).

[luckyrat]

I think I've got the Fleck2 library working now. @dlech 's fix got it going on Mono but it still wouldn't build when targeting .NET 2 so I've made some more changes (which I can't push today because git is throwing spurious out of memory errors).

It now builds when targeting .NET 3.5 but not .NET 2 (too much C#3 syntax to get that working but the resulting DLL from the 3.5 build should work in .NET 2 which is good enough for our needs).

I've built a new plgx (bundled in latest.xpi) which includes this updated Fleck2.dll and I managed to get KeePass to load it using Mono 2.10.8.1 in both .NET 4 and .NET 2 runtime modes.

I can reliably produce the black authorisation window by just starting KeePass when Firefox is already running and leaving the master password popup for my most recently used database.

I can avoid the black window problem by cancelling the master password popup (and hence allowing the main form to load) before KeeFox attempts a connection to KeePassRPC (easiest by just starting Firefox after KeePass).

@dlech if you don't get a chance to try the modal dialog trick before Saturday I'll give that a go, along with some of the other potential workarounds discussed above.

I'll also take a look at the appearance of the "authorise a new connection" window since even when the black window problem is worked around, it's a pretty poor imitation of the rendering on Windows. All the pertinent information is there but I don't think it's as clear and usable as I would like. Maybe that has been improved in later versions of Mono or maybe I can change something to get it looking better. For reference, I've attached how the dialogs appear in Mono and Windows:

[dlech]

It now builds when targeting .NET 3.5 but not .NET 2 (too much C#3 syntax to get that working but the resulting DLL from the 3.5 build should work in .NET 2 which is good enough for our needs).

Are you building using the .bat file included in Fleck2? I read something today that said the /tv command line option has no equivalent in a .csproj file, so the only way to properly build it is with the command in the .bat file.

[dlech]

@dlech if you don't get a chance to try the modal dialog trick before Saturday I'll give that a go, along with some of the other potential workarounds discussed above.

It turns out that there is a different mono bug that is causing the problem here. I had forgotten, but I ran into this before. System.Windows.Forms.Form.Invoke does not work reliably in mono. The workaround I came up with is to use a System.Windows.Forms.BackgroundWorker to invoke the main thread from other threads. In fact, when I was researching the issue originally, I read that BackgroundWorker (or System.Windows.Forms.Timer) is the preferred way to invoke the main UI thread from other threads on Windows anyway. I have sent a pull request (#217) with the workaround implemented.

So, no modal dialogs required :thumbsup:

[dlech]

I'll also take a look at the appearance of the "authorise a new connection" window since even when the black window problem is worked around, it's a pretty poor imitation of the rendering on Windows. All the pertinent information is there but I don't think it's as clear and usable as I would like. Maybe that has been improved in later versions of Mono or maybe I can change something to get it looking better. For reference, I've attached how the dialogs appear in Mono and Windows:

I had some layout issues like this in my KeeAgent addon. What I ended up doing was something like this:

public class MyForm : System.Windows.Forms.Form
...
public MyForm()
{
    InitializeComponent();
    if (Type.GetType ("Mono.Runtime") != null) {
        // make adjustments to layout need in mono
    }
}
...

[luckyrat]

Are you building using the .bat file included in Fleck2? I read something today that said the /tv command line option has no equivalent in a .csproj file, so the only way to properly build it is with the command in the .bat file.

Yeah I'm building using the batch file. I think it's normal for the /tv command line option to be specified in the .csproj file (called "ToolsVersion") although it can be overridden by /tv if needed (http://msdn.microsoft.com/en-us/library/bb383985%28v=vs.90%29.aspx). I think the problem you're referring to is that you can't load .csproj files with an old ToolsVersion into the current version of Visual Studio (it will force a project upgrade to occur) so using the command line is the only way to build using an old version of the c# compiler.

In our case, just building Fleck2 to target .net 3.5 will suffice because it doesn't use any framework libraries that were added in .net 3.0 or 3.5 (so the IL code will run in 2.0-3.5).

Interestingly, I've just discovered that version 1.2.7 does not work with .NET 2.0. That probably explains some of the mysterious "stopped working after upgrade" problems that are being mentioned.

I've just tried 1.3 in a .NET environment and seen it working all the way through the getting started tutorial so I think it's going to be reasonable to try to maintain .NET 2.0 support for 1.3 and 1.4 (which shouldn't have very much in the way of changes to KeePassRPC but will at least allow .NET 2.0 stragglers to keep using KeeFox when Firefox eventually enable their Australis changes).

[luckyrat]

I've merged 1.3 into the master branch now. Please could you take a look at the latest.xpi build and let me know if there are any outstanding issues that prevent it from working on Linux/Mac?

I'll be doing some more tests on Windows this week and hope to put a beta release onto AMO in 7-10 days if nothing serious crops up.

[luckyrat]

I did a test install on my Ubuntu VM before releasing 1.3.0b1 and it worked fine so I think we've cracked it.

Time to close down this monster; thanks for the help resolving it!

Probably best to open a new ticket for any other issues you come across during beta testing.