Codemap generic mapper does not work with UTF-16 encoded files

[TheBloke]

Hello

I just installed CodeMapper in VSCode 1.44.2 for Windows.

The extension seems to be working fine for supported languages, eg C#. But for some reason I am unable to configure the generic mapper to support a custom language.

I have tried following the example from the Wiki, and copied the example "codemap.x" configuration, renaming it "codemap.ws" as I wish to support files with a .ws extension. However my newly added variable "codemap.ws" is shown in settings.json to be invalid, and Codemap shows nothing when I test it on a sample file. I then tried copying the example in exactly, using "codemap.x", and it shows the same thing.

Here's a screenshot demonstrating my attempt to use both "codemap.ws" and "codemap.x" :

As you can see, both show as greyed out, and neither work on sample files with .ws or .x extensions.

I noticed on the Codemap extension configuration page that it has specific entries for Md, Ps1, Py and R , and if I try adding a settings.json value such as "codemap.md" or "codemap.ps1", these show as valid.

Apologies if I've done something wrong, or misunderstood something, or misread the Wiki. I am very new to VSCode.

Thanks in advance, and for creating Codemap, which looks to be exactly what I want - a way to get an outline view for a niche file extension.

PS. A separate issue, much less important - the Wiki and description talks of "Refreshing on demand via Refresh toolbar button"; I do not have any button in my Codemap bar except "Collapse All" :

[oleg-shilo]

The generic mapper may not work for two reasons:

• Invalid setting file
• invalid regex expression

So I always eliminate the first reason by adding the existing MD mapper but for a new file extension. And then opening the MD wile renamed into the extension I want.

So first ensure you are editing the right file and inserting the mapper value in the right place:

I suggest you also do it for the MD mapper (i have attached the file). settings.zip

When you place the file in the C:\Users\<user_name>\AppData\Roaming\Code\User folder you can test it by opening any MD file renamed in the *.ws:

When you get it working you can start experimenting with your regex as your infrastructure is already working.

========================

And I will have a look at the missing refresh button. Txs

[oleg-shilo]

The new issue has been raised: https://github.com/oleg-shilo/codemap.vscode/issues/34

And fixed now.

[TheBloke]

Thanks very much for the fast reply, and for fixing issue #34.

OK, so I understand a lot better now. Firstly, I realise now that it is not a problem that "codemap.ws" shows as 'invalid' in settings.json, I can ignore this.

So the issue I have is that Codemap will for some reason not recognise the files I am trying to work with.

EDIT: I have identified the problem; my .ws files have UTF-16 LE encoding, and this does not seem to work with Codemap? If I re-save the files as UTF-8, it works immediately. See next comment.

First, here's a working example: I followed your suggestion and copied "codemap.md" settings to "codemap.ws", and then renamed a .md file to .ws. This worked fine. Then I edited the settings to include a couple of simple regexes, and confirmed this also worked fine in my test file, as shown here:

• In right hand split, settings.json with some test config
• In middle split, test file (started out as MD file then renamed to .ws, and test enum/functions added)
• On left, Codemap is recognising the function and enum entries from this test file.

So that's all perfect.

But then when I try with a real .ws file, I get nothing:

So I am thinking there must be some issue specific to the files I'm trying to load?

Could it be because these files are in UTF-16 format? I don't know if that affects your code, but I do know I had trouble when I was trying to grep these files on a Solaris system, due to the unusual file encoding.

Below I have attached one of the files I am trying to work with. Would you be able to have a look to see whether it works for you, and if not, what the problem might be?

playerTypes.ws.zip

And here is the basic config that I am testing with, which should pick out the enum and function definitions from that file:

"codemap.ws": [
        {
            "pattern": "function \\w*",
            "clear": "function",
            "prefix": "",
            "icon": "function"
        },
        {
            "pattern": "enum \\w*",
            "clear": "enum",
            "prefix": "",
            "icon": "function"
        }
    ]

Thanks so much again for your time in helping me.

[TheBloke]

Could it be because these files are in UTF-16 format? I don't know if that affects your code, but I do know I had trouble when I was trying to grep these files on a Solaris system, due to the unusual file encoding.

This is the problem!

I just saved one of my .ws files as UTF-8, and immediately Codemap worked with it:

So, is there any way that Codemap could support UTF-16 LE files? Unfortunately I have to work with this encoding, as I am writing modifications to existing code, and all the files are UTF-16 LE.

Thanks again.

[oleg-shilo]

Can you check this patch before I publish it. It should fix your problem:

[Uploading codemap-1.11.2.vsix.zip…]()

[TheBloke]

Thanks very much for making the fix. I'd be glad to test it.

The ZIP doesn't seem to be downloadable at the moment? Did it upload OK? For me it shows like this:

[TheBloke]

I've grabbed the updated file from the source repo, going to try putting it in place now. One minute..

[TheBloke]

It works! I now have enums and functions visible on my UTF-16 .ws files.

I got a few errors when I manually compiled mapper_generic.ts to mapper_generic.js, but the resulting .js file still seems to work fine (and I'm sure these errors are just because I manually compiled it in out/src)

So yes, please could you push this to an official release.

Thanks very much for the fast resolution!

[oleg-shilo]

Great. Will publish in a min

[oleg-shilo]

Done