OpenPGP.js and LiquidCore in Android

[boby1975]

Hi guys, I use LiquidCore (Node.js virtual machine) inside my Android App (java). And I'm trying to use openpgp.js under Node. But it doesn't work. The problem is described in more detail here https://github.com/openpgpjs/openpgpjs/issues/1080

How to convince LiquidCore build system to treat openpgp.js as an external dependency and require openpgp.js at runtime, instead of at build time?

Unfortunately I could not get openpgp.js outside of the bundle (as an external dependency) during apk building in Android Studio. And the file "metro.config.js" with "...blacklistRE: blacklist([/dist/.*/]" and other options didn't help me.

Thanks a lot

[ericwlange]

Hi @boby1975 ,

You will have to hack around it. It looks like openpgp is using its own bundled code, and then LiquidCore is bundling on top of it. I have managed to work around it by replacing offending require() statements with LiquidCore.require(). This bypasses the bundler override and uses stock node require().

Also, for whatever reason, they have chosen not to package asn1.js into openpgp.js. I am guessing they had a problem with packaging it due to some circular references or something. So it is looking for that module at runtime. I already built some intelligence into the LiquidCore bundler so that it can handle circular requires.

To get this to work, you need to add to the top of your javascript file:

const asn1 = require('asn1.js')
global.requireASN1 = () => asn1
const openpgp = require('openpgp')

It is important that you create the requireASN1() function before you require openpgp.

Then you must patch your node_modules/openpgp/dist/openpgp.js file:

8016c8016
<     var crypto = require('crypto');
---
>     var crypto = LiquidCore.require('crypto');
23792c23792
<     crypto = require('crypto');
---
>     crypto = LiquidCore.require('crypto');
23821c23821
< const NodeReadableStream = _util.isNode && require('stream').Readable;
---
> const NodeReadableStream = _util.isNode && LiquidCore.require('stream').Readable;
24139c24139
< const NodeBuffer = _util.isNode && require('buffer').Buffer;
---
> const NodeBuffer = _util.isNode && LiquidCore.require('buffer').Buffer;
24596c24596
< const NodeReadableStream = isNode && require('stream').Readable;
---
> const NodeReadableStream = isNode && LiquidCore.require('stream').Readable;
30022c30022,30023
< const asn1 = nodeCrypto ? require('asn1.js') : undefined;
---
> //const asn1 = nodeCrypto ? require('asn1.js') : undefined;
> const asn1 = nodeCrypto ? requireASN1() : undefined;
30622c30623,30624
< const asn1 = nodeCrypto ? require('asn1.js') : undefined;
---
> //const asn1 = nodeCrypto ? require('asn1.js') : undefined;
> const asn1 = nodeCrypto ? requireASN1() : undefined;
31185c31187
< const nodeCrypto = _util2.default.detectNode() && require('crypto');
---
> const nodeCrypto = _util2.default.detectNode() && LiquidCore.require('crypto');
43138c43140
<     return require(module);
---
>     return LiquidCore.require(module);

I was able to get it to generate a key with the following code:

const asn1 = require('asn1.js')
global.requireASN1 = () => asn1
const openpgp = require('openpgp')

var options = {
  userIds: [{ name: 'Alice', email: 'alice@example.com' }],
  numBits: 2048,
  passphrase: 'secret'
}
var pubKey, privKey
openpgp.generateKey(options).then(key => {
  privKey = key.privateKeyArmored
  pubKey = key.publicKeyArmored
  console.log('Key generated')
  console.log('Public key = ' + pubKey)
})

2020-04-17 12:03:22.833 9200-9252/org.liquidplayer.testapp I/stdout: Key generated
2020-04-17 12:03:22.833 9200-9252/org.liquidplayer.testapp I/stdout: Public key = -----BEGIN PGP PUBLIC KEY BLOCK-----
    Version: OpenPGP.js v4.10.2
    Comment: https://openpgpjs.org

    xsBNBF6ZfukBCAC96Fk1Z+dD0CMOAzL0OSewNovtQVDmpb+9SHEjB4LdhH5o
    obreI2Ua7+MrJkg6K06C7ED21qKI1+HDXtiTUALj1ZsCKxcuy3lx8ZjJNo+d
    4WrcFarPBL3KOe8aSjYKqMzrlM4MzXWbfb5W7gUxGExILuuU8a9GkeshtgFE
    8l6/rZwdiAm7FcWAwZC1geZdIw/xu7tNKZ1PAtzFPHxioxHpsMoXS/4d8w/L
    BjJDqfWWEERLZ+EK12vwX3Ftxh6OWsfC5Oeb9B/JSKC8YoDpV3zG1DDUiXF6
    yx7Zp/Pk7feKLZI1tPnh/EyHFehNylHcltF8ErgEfsVl83vzMH3VQU8DABEB
    AAHNGUFsaWNlIDxhbGljZUBleGFtcGxlLmNvbT7CwHYEEAEIACAFAl6ZfukG
    CwkHCAMCBBUICgIEFgIBAAIZAQIbAwIeAQAKCRCxwSPraGOBAN87B/9kFRf1
    ZA65mwAWCwnGuoFX425pcP08b9Kquykx9KKYpt2I5/NbCIBdGFbYA0VPJMV/
    /h6Lwr2wqEc3MR4uoKm8EQReMEp2MdrxJunwVC+2j7OTngxz6EKBpgxSkYos
    KlFH+VmTtKtoY8sQ8NhgUpumRhC1c2Gg8Kp00A+zXmgwA1GN3+4pEB0uiQtT
    Cjls3TJT1xTRMhR2GWM96y+zSmiAMofQOOBnPlnIq9Bxl96ylaOo13h/Q75i
    BAplNVOX8j59eET/o4G8cG2uG715SV/RJ7MRJxCEgijnIZXmAr5FPtKQ+OUV
    QqOAPIIJE4bn3zkBiLVcpG61Fk7GfeWk+5YXzsBNBF6ZfukBCADclvmuQUa2
    F4A8qromgtN9ayyn4SKX588YSF6F/ZGFqK6f5Q2fb01k8hgwpOoW64YjfOQj
    nO5Ub3tnxes5dIwOOKXktT1bbf9DOLISH9Wip1rER3cVdSTuYxN98oJBiZpd
    xqAAWo8vU7REAw7wN0bRpSpZRlYT1GK7nYBid07oS5rEHxAFzfQN97Ih2gEW
    kaH2LHuXD/5zhoi3J2ulNoZ0AvjznFCtWV7idiBFdXMyEyRMHXNUTbd81SmF
    IRkD28iUxMzzt37o3cmqnN6/cq8+SM0crxhibRhFwYRuLzZeMBJLZ9+0aQO3
    1Ylrop01yQ0zj4hYoCExg86qcKgtzINLABEBAAHCwF8EGAEIAAkFAl6ZfukC
    GwwACgkQscEj62hjgQBPhQf/TW5kvv5JI/8kpuRs7lvWeHED/qVcwCDeTGF4
    96fH8zXsi1/cz/c9ntXSeBD6oB91d53GfQLCwQSx9eDLo/UG6T4IWfDQvMB3
    l8q4Hgc6ynQsm3qMANZdMSvQ3hS16bR70vCkYCPSyuG1rZ1o8XUp3R1Be5l0
    1IZHJBMkFbpTyNwHtBvq5WIDAVi+Cm8Mv1JmX4STbS5tRs3yNpK/ASIEmm6W
    tGJMOhjLxAGhwyMHBIKLhjEHEiaSHYlEDiXDwaZ4GWg/5qKBjSW0X/aDlpNf
    C6/0thIGSt6P4WnN6LXxBg5NL1PWNdK9IVa9B8pDbL/dpMZcJXCFuMCsKZEC
    gIn9Mg==
    =X7se
    -----END PGP PUBLIC KEY BLOCK-----

It is possible that I missed other require statements that need to be hacked, but you will discover that through testing. Finally, if you want your new openpgp.js file to work in regular (non-LiquidCore) node, you will have to add some logic to detect the presence of global.LiquidCore before using it. But I leave that up to you.

Hope this helps.

[boby1975]

Now it works. Thank you very much. I'm not so strong in JS, so you saved me a lot of time.

[ArtemisKS]

Hello @ericwlange ! Thanks for your very detailed and practically derived answer, but I exactly repeated your steps, and, alas, it didn't work. It's worth to mention that I'm deploying LiquidCore on iOS, and while my colleague replicated your exact steps on Android and it worked, on iOS it didn't. Although before I was able to run it, only to get exceptions when trying to work with OpenPGP methods later. An exception which pops in runtime now, just after LiquidCore server is started, is this: Error: TypeError: undefined is not an object (evaluating 'superCtor.prototype'), which occurs in this chunk of code:

 module.exports = function inherits(ctor, superCtor) {
    ctor.super_ = superCtor
    ctor.prototype = Object.create(superCtor.prototype, {
      constructor: {
        value: ctor,
        enumerable: false,
        writable: true,
        configurable: true
      }
    });
  };

I've been trying to deal with it for a while now, but to no avail. I actually switched to node version v10.15.3, as in LiquidCore, then updated LQ to v0.7.7 (previously I was working on 0.7.3), but, as I suspected, it didn't help. I assume it's the problem of a similar kind, as on Android, because, say, npm thinbus-srp works well on LQ & iOS. By the way, I use Xcode 11.3.1. I also noted that resulting bundled files are different on iOS & Android, given the same example.js file, and I'm guessing that it's not the way it should be. I build all LiquidCore dependencies precisely as described in docs, thus I can't figure out, where the problem lies. I would be very grateful if you could help me out here or at least give a nudge in the right direction! If you need any of the files I use for building LQ, I would be glad to provide them.

[ericwlange]

Hi @ArtemisKS . There are some bugs in iOS, it seems. There isn't a workaround at the moment. I fixed some of the issues on my local machine, but some crypto functionality doesn't seem to work properly yet. I will have to dig deeper.

[ArtemisKS]

Thanks for your efforts @ericwlange ! I hope you do manage to solve this issue, as it’s really vital for my work! If I may be of any assistance, of any kind, please write me whenever you need.

[ArtemisKS]

Hey again @ericwlange ! Sorry to bother you still, but if we can maybe keep somehow in touch, that would be awesome. Because, you know, the solution of this problem is really pivotal for the project I’m working on, and I haven’t been able to solve it on my own, having a pretty limited knowledge of JS and Node.js in particular. I’d like to reiterate, that I would be extreme grateful if you help to solve this issue, which, I guess, you most surely can (because I can not). I fully understand that it may require some of your time, and in addition to my gratitude as it is, would be willing to reimburse LiquidCore itself, not only from a personal view, but considering its overall great practical value and potential. Hope for your understanding, and in case you’d like to contact me, not just in the current thread, here’s my email: artemkupriyanets@yahoo.com

[ericwlange]

Hi @ArtemisKS The issue is not yet resolved. I found the problem that was causing crypto to fail (TypedArray.for() and TypedArray.of() were not polyfilled properly). But once I fixed that, a memory error or buffer overrun of some sort is occurring causing the process to crash. It is very difficult to pin down the cause because it is not deterministic.

I am still working on it, but I can't give a timeframe at the moment because I don't know the root cause yet. Also, LiquidCore is a passion project for me, but not my day job, so I only have limited windows of time to devote to it.