Problem with serialport module

[vankxr]

I am trying to get SWO output with OpenOCD using an FT2232H chip as a debugger. I am using the second interface of that chip as a fast UART to receive ITM data.

Currently, this extension reports a problem with serialport not being loaded due to a VSCode update. After some research I found out that I need to update the binary_modules. My VS Code is running on Node 12.4.0 and Electron 6.1.5, which is way newer than the version provided.

I tried updating my binary_modules but without success. The best I could get was NODE_VERSION 72, and it requires 73, I assume it has something to do with the Electron version.

Can anyone update the binary_modules or give me advice on how to do it myself?

Thank you

[HoriLiu]

Maybe you can refer this issue: permanent Fix for the serialport problem

[haneefdm]

OMG, @Marus would probably so glad if it works.

I try to contribute but this is something I don't know how to do and sounds like a super hassle.

[vankxr]

I managed to generate a compatible version of the serialport module, using NVM, installing electron 6.1.5 and electron-rebuild, but there was another problem preventing this line of code from working properly, due to webpack.

After packing the code into the dist/extension.js file, the module global variable was being shortened and replaced by a local copy. When the binary_modules normalized path was being spliced, it was infact being spliced into the local copy instead of the global module search path, preventing the next require from finding the serialport module.

I fixed it by creating a symbolic link inside the dist folder, like so: ln -s ../binary_modules/v12.4.0/linux/x64/node_modules node_modules since that folder is already part of the module search path

(opening extension.js with an editor, searching for that part of the code and replacing the local copy, generated by webpack, with module.path also works)

It's a ghetto fix, but it's working now.

[haneefdm]

@vankxr Could you help me go through the exact steps you went through to create a new version? Maybe I can push an updated serialport with the next release that was supposed to be this week. I will see about automating it for all platforms.

This is an old issue #85

[haneefdm]

I think I figured it out. I created a document for future reference on how to do it and also added a unit test.

@Marus, I hope you can answer this.

The darwin directory with v8.9.3 was 3232K, but with v12.4.0 it is 5436K. I am not sure why it is so much bigger. Is this okay? I have already pruned it for production purposes. The new modules are in v12.4.0

buffer-alloc
buffer-alloc-unsafe
buffer-fill
isarray
os-homedir
readable-stream
string_decoder
to-buffer
xtend

Also, I am not sure how this works. About dependencies, buffer-alloc for is also a dependency for Cortex-Debug, so shouldn't we find it already because it would be in the module-path?

Also, we add the binary_modules to the beginning of the path array which means it now overrides modules the system already has?

[vankxr]

@haneefdm I can check my bash history, but basically I installed NVM so I could easily select the NodeJS version I wanted (the one my VSCode was running). Then I installed the Electron version my VSCode was using, with npm i -g electron@6.1.5, and also npm i -g electron-rebuild After that I created a "dummy" package.json by doing npm init and accepting the defaults with Enter until the end. After that I installed node-serialport, with npm i --save serialport, and finallly electron-rebuild --module-dir ..

At the end I was left with a node_modules folder containing the correct binding binaries corresponding to NODE_VERSION 73 (aka NodeJS 12.4.0 and Electron 6.1.5).

I just deleted the package.json file as it was only used to save serialport so electron-rebuild could know what to rebuild, and then I copied the node_modules folder to ~/.vscode/extensions/marus25.cortex-debug-0.3.4/binary_modules/v12.4.0/linux/x64/.

After that I cd ~/.vscode/extensions/marus25.cortex-debug-0.3.4/dist and ln -s ../binary_modules/v12.4.0/linux/x64/node_modules node_modules

[haneefdm]

@vankxr My process was a bit different. I started from a different repo which avoided the global install of electron. I was missing the dummy project part but pruning probably did the same thing. Following is my doc.

Serial Port Binary Module

Manual creation of serialport binary dependency. Intended for developers and not end-users.

• Clone the serial port

git clone https://github.com/serialport/electron-serialport.git
cd electron-serialport

• Edit package.json to the particular electron version we want. Only edit the electron package. Leave other packages the way they are.

"devDependencies": {
    "electron": "6.1.5",
    ...
},

• Run the following commands

npm install
# A window opens and gives you the exact <node-version> that electron is built with.
# You will need this later.
npm start
# get rid of development dependencies
npm prune --production`
# No you have a `node_modules` directory that is ready to be deployed

Depending on the OS/platform/etc., copy/replace the node_modules directory to the appropriate location in the cortex-debug repo

• MacOS: binary_modules/<node-version>/darwin/x64/node_modules

[haneefdm]

Now I want to see if I can automate this.

[vankxr]

But isn't there a way of properly installing the module when VSCode installs the extension? I mean, if serialport is a problem because it needs native bindings, then all modules that do require that will be a problem...

[haneefdm]

It is pretty risky to do it at the user desktop, lots of things can fail. It is my end goal though.

The serial port module has to be compiled and linked with the exact electron version needed. C++ compilation is involved so. The electron version of VSCode does not change very frequently. Having said that, VSCode is about to bump electron to 6.1.6 with plans to move to 7. When they first moved to 6, they had hundreds of problems.

If I can't automate it, I can't get to the end goal either. I am taking baby steps. If it was easy, @Marus would have done it a long time ago. If you have ideas, let me know.

[vankxr]

I understand that compilling at the user desktop can be risky. But this is not sustainable IMO, even though VSCode only bumps Electron/Node versions from time to time.

It has to exist a way of dealing with Node modules with native bindings, like prebuild-install

[Marus]

Yes - binary modules have been a pain since the beginning of the project.

Currently we don't have the ability to install node modules at install/run-time - only at build-time; simply a limitation of VSCode (there had been some talk in the past of enabling this - but has gone nowhere to my knowledge). And of course since we are talking binary modules we can't have a single copy that covers all the platforms that the extension is used on. Which is why currently I have to manually build and bundle (and have special loading code for) the binary modules - as I have to bundle multiple copies (at least 32 bit is dying out so it is a few less variants that need to be included now).

I have been investigating some things to at least automate the building of the modules - to make my process simpler - and do have some thoughts on even possible ways so they wouldn't have to be pre-bundled (although that's less important to me).

But if you have thoughts - or something working for the automation of building the modules on all platforms involved - then would love to see that as well.

Unfortunately something like prebuild-install doesn't work easily either - as there isn't a good way to build them cross platform for Linux, Windows and macOS all on one machine (at least not easily).

[haneefdm]

@Marus I am pretty close to a script to create the serialport binary module. I need some more time. I know @Marus will find a better way. I still think we have to build, deploy and release for the basic install.

One doubt I had was if we have to support previous releases of vscode/node/electron. I am unclear about VSCode roadmap wrt. electron and how far back we have to maintain compatibility

Finally, I don't want to do it at the user's machine during or after the extension install/activation. Sooo many things can go wrong in an uncontrolled environment and not everyone needs it.

Thoughts?

[vankxr]

I don't see why would it be a problem to "force" users to update to latest VSCode to keep using this extension (thus removing the need to worry about previous versions).

I personally never encountered anything worth keeping VSCode from updating.

By the way, any thoughts on the problem webpack is generating? I talked about it earlier, didn't you encounter it while updating your binaries? I am pretty sure it's an easy fix, just a matter of preventing it from shortening the global var "modules" or something.

[haneefdm]

I am wary of disturbing someone's workflow in the case they are relying on it for one reason or another. I am paranoid that way.

I personally never encountered anything worth keeping VSCode from updating.

I have. When they switched to a new version of Electron.

By the way, any thoughts on the problem webpack is generating? I talked about it earlier, didn't you encounter it while updating your binaries? I am pretty sure it's an easy fix, just a matter of preventing it from shortening the global var "modules" or something.

Not sure what you are referring to. More details? I am sorry I missed any earlier comments.

[Marus]

My loading code does support having multiple sets of binary modules (the version is in the paths so will load the correct one) - so in general when I have been updating them I've kept the current one and the previous version. If they're any further back than that - :shrug:

As for how you were building it Haneef - you don't actually really need modify the serial port packages like it looks like you were doing. You can actually do it all through environment variables:

This is approximately the script I was using for building the various binary modules that were used/planned to be used.

#!/bin/sh

export npm_config_arch=x64
export npm_config_build_from_source=true
export npm_config_disturl=https://atom.io/download/electron
export npm_config_runtime=electron
export npm_config_target=$1
export npm_config_target_arch=x64

HOME=$(pwd)/.electron-gyp-64-$1 npm install serialport usb

You pass in the electron runtime version in as the first parameter.

I actually have updated versions of the Linux and macOS binary modules already prepared - the only reason I haven't finished with the current update and released it is that I am currently lacking a functional Windows box to get the modules for Windows.

[vankxr]

@Marus I tried doing something like you said, with environment variables, but wasn't able to compile for a specific electron version.

@haneefdm webpack is shortening the global variable modules, so when the code tries to add the path to the binary_modules directory, it actually adds it to the local copy, instead of the global array, and then the module.require call does not work because it finds no serialport since the search path is not there.

[Marus]

@vankxr - what platform/OS were you trying to do the build on? I know that method still works fine on Linux and macOS (as I just built the 6.1.5 modules right then).

The webpack problem is also interesting - looks like it is rewriting the module.require into using web packs require (instead of native node/electron require). I expect I can come up with a fix - will try that tomorrow.

[haneefdm]

@Marus your script does work. It does link/bind to the specific version of electron. Thanks. I noticed you added 'usb'. Is that a new thing we will need. I don't remember that earlier.

I have a script that also takes the electron version as the argument except I did not have all the npm_config magic so, I used the package.json method. Again, thanks

I have a 64-bit Windows 10 machine but no 32-bit ones. Wonder if I can cross-compile

[Marus]

At least on linux you used to be able to do cross to x86 from an x64 machine - and I believe I did that on Windows as well in earlier versions.

The usb module isn't actually required for current releases; that was in there as I had been experimenting with adding native SWO/ITM output support for the black-magic-probe. But none of that has been released - so at the moment usb could be left off that command.

If you can actually build the windows modules and zip the node_module directories up for me I can try to get the webpack require issue solved and see if we can get an update to this actually out the door - even if it isn't a full solution for the future.

[haneefdm]

I am on macOS and where I tried the script. Looks like @vankxr is on either a Linux or a Mac. Not sure.

I will try Linux tomorrow and Windows after that.

[Marus]

I know it works fine on macOS and (Ubuntu 18.04) Linux - as I've done it there myself.

For the cross compiling you have to replace x64 with x86 in the npm_config_target_arch variable (or maybe both npm_config_target_arch and npm_config_arch)

[haneefdm]

Yup, that's what I thought. Windows, I wasn't sure about and what tools I have to have pre-installed. Will find out once I get there.

[Marus]

Ok - see if you can make the windows modules (worse comes to worse we could release with all platforms other than Windows/x86) and send them to me as a zip I'll do an update - think I have the fix for the webpack issue we're seeing - just need to test it out a little bit.

You can fell free to do it without the usb ones at the moment - as those aren't currently needed. If I ever get to finishing that BMP stuff I was working on hopefully by then I've got to actually fixing my Windows computer and I'll be able to do the build of those modules by myself.

[vankxr]

I am using Linux.

[haneefdm]

While we were talking, VSCode just updated and Electron version now is 6.1.6, but still uses Node 12.4.0

Btw, I have a script that now generates the serial port module into the binary_modules directory.

# Takes two args. Electron version # and Node version #
./serial-port-build.sh 6.1.6 12.4.0

https://github.com/haneefdm/cortex-debug/blob/master/serial-port-build.sh

So far, it works on macOS, Linux 64-bit. On Linux, I generate both 32-bit and 64-bit modules. I still have to test it on 32-bit Linux using modules generated on a 64-bit machine. Also, the usb fails on Linux, so I removed it from my script. Next is to venture into Windows land. Dreading it.

[haneefdm]

Beating my head on 32-bit Linux. I wish I had known. Neither Chrome or VSCode even has a 32-bit Linux version anymore!?!? I did not know. Spent hours setting up a Linux 32-bit VM only to find out that.

Am I wrong?

[haneefdm]

One more little step. It works on Win 64-bit. Generation of the module works, Unit test passes which loads the module and lists the serial ports available successfully.

Problem w/ Windows: No freaking idea why it worked. I just ran the script (had to unix2dos it) and it just worked. My Visual Studio setup must have been just right. I did not do anything special It just worked on the first try. My BIG problem is how someone else duplicates what I did. My PC dies, I get hit by a bus.

I updated the scripts accordingly in my fork.

Questions? Concerns?

In my checklist, I got the following macOS: Done Linux 64-bit: Done. Only tested Ubuntu though Linux 32-bit: Not supported. Windows 64-bit: Done? see above ** Windows 32-bit: Pending. Logistical issues. Got no licenses

[Marus]

That looks great.

Yeah, they dropped 32 bit on Linux a while back - so don’t have to deal with that any more. Still feel like you should be able to do 32-bit cross compiles for Windows (even if you can’t do a proper test). That one I don’t feel to bad about not supporting mind you - can’t imagine there is much usage of 32 bit windows (honestly I’m a bit surprised VSCode still offers 32 bit on Windows)

The electron updates do seem to be coming more frequently now - so getting this even partially automated is a big win in my mind.

I’ve got a Windows VM mostly setup now as a stopgap until I get around do replacing my dedicated windows box - so will try to finish that up tomorrow and try the script on that vm. What I’m hoping to eventually do if we have this script working is to tie it until my CI build server so can have it pretty much be a 1 click update.

[Marus]

And in regards to it just working - when I first build the binary modules I didn’t have any issues getting them on Windows either (much to my surprise at the time - usually find Windows to be a pain for most things) - so would mostly be a matter of making the scripts work.

[haneefdm]

I tried 32-bit windows compile. Failed. Will investigate later

[haneefdm]

I made a PR #245 with the new serialport added. I think we can close this once it is released

[Marus]

@vankxr - I uploaded a pre-release to the GitHub releases page - v0.3.5-pre1 - this should have the new serialport modules as well as a fix for the loading after webpack processing has been completed.

Should hopefully work on macOS, Linux and Windows (64bit only).

[Marus]

What arch did you specify when you tried to cross-compile for 32-bit windows. I believe it is actually ia32 you have to specify not x86 like I previously mentioned if you were trying that.

[haneefdm]

arch = x64 os = win32

I did not use ia32 or x86

On Sun, Feb 9, 2020 at 6:24 PM Marcel Ball notifications@github.com wrote:

What arch did you specify when you tried to cross-compile for 32-bit windows. I believe it is actually ia32 you have to specify not x86 like I previously mentioned if you were trying that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Marus/cortex-debug/issues/243?email_source=notifications&email_token=AJ23ST2MNH2WHTK44HG3P73RCC3GLA5CNFSM4KQMJL32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELHA6HA#issuecomment-583929628, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ23STZKS5PXRPXSRL65NY3RCC3GLANCNFSM4KQMJL3Q .

[haneefdm]

Oh, you mean for 32-bit binaries that did not work. Yes, I had tried x86. I can quickly try ia32 if you want.

On Sun, Feb 9, 2020 at 6:38 PM Haneef Mohammed haneefdm@gmail.com wrote:

arch = x64 os = win32

I did not use ia32 or x86

On Sun, Feb 9, 2020 at 6:24 PM Marcel Ball notifications@github.com wrote:

What arch did you specify when you tried to cross-compile for 32-bit windows. I believe it is actually ia32 you have to specify not x86 like I previously mentioned if you were trying that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Marus/cortex-debug/issues/243?email_source=notifications&email_token=AJ23ST2MNH2WHTK44HG3P73RCC3GLA5CNFSM4KQMJL32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELHA6HA#issuecomment-583929628, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ23STZKS5PXRPXSRL65NY3RCC3GLANCNFSM4KQMJL3Q .

[haneefdm]

Yes, ia32 works. At least it compiled fine.

I can do a PR if you like.

[image: image.png]

On Sun, Feb 9, 2020 at 6:39 PM Haneef Mohammed haneefdm@gmail.com wrote:

Oh, you mean for 32-bit binaries that did not work. Yes, I had tried x86. I can quickly try ia32 if you want.

On Sun, Feb 9, 2020 at 6:38 PM Haneef Mohammed haneefdm@gmail.com wrote:

arch = x64 os = win32

I did not use ia32 or x86

On Sun, Feb 9, 2020 at 6:24 PM Marcel Ball notifications@github.com wrote:

What arch did you specify when you tried to cross-compile for 32-bit windows. I believe it is actually ia32 you have to specify not x86 like I previously mentioned if you were trying that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Marus/cortex-debug/issues/243?email_source=notifications&email_token=AJ23ST2MNH2WHTK44HG3P73RCC3GLA5CNFSM4KQMJL32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELHA6HA#issuecomment-583929628, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ23STZKS5PXRPXSRL65NY3RCC3GLANCNFSM4KQMJL3Q .

[haneefdm]

Just did a PR in case you wanted it.

On Sun, Feb 9, 2020 at 6:43 PM Haneef Mohammed haneefdm@gmail.com wrote:

Yes, ia32 works. At least it compiled fine.

I can do a PR if you like.

[image: image.png]

On Sun, Feb 9, 2020 at 6:39 PM Haneef Mohammed haneefdm@gmail.com wrote:

Oh, you mean for 32-bit binaries that did not work. Yes, I had tried x86. I can quickly try ia32 if you want.

On Sun, Feb 9, 2020 at 6:38 PM Haneef Mohammed haneefdm@gmail.com wrote:

arch = x64 os = win32

I did not use ia32 or x86

On Sun, Feb 9, 2020 at 6:24 PM Marcel Ball notifications@github.com wrote:

What arch did you specify when you tried to cross-compile for 32-bit windows. I believe it is actually ia32 you have to specify not x86 like I previously mentioned if you were trying that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Marus/cortex-debug/issues/243?email_source=notifications&email_token=AJ23ST2MNH2WHTK44HG3P73RCC3GLA5CNFSM4KQMJL32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELHA6HA#issuecomment-583929628, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ23STZKS5PXRPXSRL65NY3RCC3GLANCNFSM4KQMJL3Q .

[haneefdm]

https://github.com/Marus/cortex-debug/pull/247

On Sun, Feb 9, 2020 at 6:46 PM Haneef Mohammed haneefdm@gmail.com wrote:

Just did a PR in case you wanted it.

On Sun, Feb 9, 2020 at 6:43 PM Haneef Mohammed haneefdm@gmail.com wrote:

Yes, ia32 works. At least it compiled fine.

I can do a PR if you like.

[image: image.png]

On Sun, Feb 9, 2020 at 6:39 PM Haneef Mohammed haneefdm@gmail.com wrote:

Oh, you mean for 32-bit binaries that did not work. Yes, I had tried x86. I can quickly try ia32 if you want.

On Sun, Feb 9, 2020 at 6:38 PM Haneef Mohammed haneefdm@gmail.com wrote:

arch = x64 os = win32

I did not use ia32 or x86

On Sun, Feb 9, 2020 at 6:24 PM Marcel Ball notifications@github.com wrote:

What arch did you specify when you tried to cross-compile for 32-bit windows. I believe it is actually ia32 you have to specify not x86 like I previously mentioned if you were trying that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Marus/cortex-debug/issues/243?email_source=notifications&email_token=AJ23ST2MNH2WHTK44HG3P73RCC3GLA5CNFSM4KQMJL32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELHA6HA#issuecomment-583929628, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ23STZKS5PXRPXSRL65NY3RCC3GLANCNFSM4KQMJL3Q .

[vankxr]

I didn't test the pre-release yet since I ended up compilling serialport on my own, but will give it a try once I update my VSCode

[Marus]

Merged in that PR for the ia32 (I expect it will work even if we don't have the ability to test it) and the webpack loader fix as well @haneefdm.

[diegoherranz]

@vankxr - I uploaded a pre-release to the GitHub releases page - v0.3.5-pre1 - this should have the new serialport modules as well as a fix for the loading after webpack processing has been completed.

Should hopefully work on macOS, Linux and Windows (64bit only).

I've just tested v0.3.5-pre1 installing the vsix file manually and I still get the same error:

Unable to load Serial Port Module. A recent Visual Studio Code update has likely broken compatibility with the serial module. Please visit https://github.com/Marus/cortex-debug for more information.

Should that work? Or is not expected to work until there is a formal release?

Info:

Version: 1.42.0
Commit: ae08d5460b5a45169385ff3fd44208f431992451
Date: 2020-02-06T10:51:23.649Z
Electron: 6.1.6
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Linux x64 4.4.0-173-generic

Thanks!

[Marus]

@diegoherranz and/or @vankxr Can you try the v0.3.5-pre2 - that has a new change that I think should fix the webpack issue.

[vankxr]

@Marus Just updated my VSCode, uninstalled 0.3.4 and installed 0.3.5-pre2 Tested with this configuration: { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Debug", "cwd": "${workspaceRoot}", "executable": "./bin/v1.icyradio-dsp.elf", "request": "launch", "type": "cortex-debug", "preLaunchTask": "Build", "servertype": "openocd", "configFiles": [ "${workspaceRoot}/.vscode/openocd-scripts/target.cfg" ], "device": "ATSAMV71Q21B", "svdFile": "${env:ARM_LIBROOT}/Atmel.SAMV71_DFP.2.4.182/samv71b/svd/ATSAMV71Q21B.svd", "interface": "swd", "swoConfig": { "enabled": true, "swoFrequency": 12000000, "cpuFrequency": 300000000, "source": "/dev/ttyUSB1", "decoders": [ { "port": 0, "label": "Console", "type": "console", "showOnStartup": true }, ] } } ] } Working as expected

[Marus]

@haneefdm - since it worked for @vankxr - I'll merge that PR in and you can include it with your next release. Then we can close this ticket.

[haneefdm]

Thanks. Yes, I just saw that.

[diegoherranz]

@diegoherranz and/or @vankxr Can you try the v0.3.5-pre2 - that has a new change that I think should fix the webpack issue.

I've tested v0.3.5-pre2 and it works perfectly. Thanks for all the work!

Version: 1.42.1
Commit: c47d83b293181d9be64f27ff093689e8e7aed054
Date: 2020-02-11T14:50:36.977Z
Electron: 6.1.6
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Linux x64 4.4.0-174-generic