Cannot generate lib/lexer.js with emscripten 1.40.1

[richardo2016]

I cannot build lib/lexer.js by myself with make lib/lexer.js,

Error like below:

$ make lib/lexer.js
emcc ./src/lexer.c -o lib/lexer.js -s WASM=0 -Oz --closure 1 \
        -s EXPORTED_FUNCTIONS=["_parse","_sa","_e","_ri","_re","_is","_ie","_ss","_ip","_se","_ai","_id","_es","_ee","_f","_setSource"] \
        -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s SINGLE_FILE=1 -s TOTAL_STACK=15312 -s --separate-asm
emcc: warning: --separate-asm works best when compiling to HTML.  Otherwise, you must yourself load the '.asm.js' file that is emitted separately, and must do so before loading the main '.js' file. [-Wseparate-asm]
emcc: error: cannot --separate-asm when emitting wasm, since not emitting asm.js
make: *** [lib/lexer.js] Error 1

---

Env Info

OS: macOS emscripten version: 1.40.1

as mentioned #86 , I installed emscripten 1.40.1 and activate it in my bash session.

$ ./emsdk install 1.40.1
$ ./emsdk activate 1.40.1
$ source ./emsdk_env.sh

I'm sure I haved installed it, because I checked it by ./emsdk list and ./emcc -v.

$ ./emsdk list
./emsdk list

The *recommended* precompiled SDK download is 2.0.30 (c69458f1bbf3ef5b8da4e934de210659cc9bca04).

To install/activate it, use one of:
         latest                  [default (llvm) backend]
         latest-fastcomp         [legacy (fastcomp) backend]

Those are equivalent to installing/activating the following:
         2.0.30             
         2.0.30-fastcomp    

All recent (non-legacy) installable versions are:
         2.0.30    
         2.0.29    
         2.0.28    
         2.0.27    
         2.0.26    
         2.0.25    
         2.0.24    
         2.0.23    
         2.0.22    
         2.0.21    
         2.0.20    
         2.0.19    
         2.0.18    
         2.0.17    
         2.0.16    
         2.0.15    
         2.0.14    
         2.0.13    
         2.0.12    
         2.0.11    
         2.0.10    
         2.0.9    
         2.0.8    
         2.0.7    
         2.0.6    
         2.0.5    
         2.0.4    
         2.0.3    
         2.0.2    
         2.0.1    
         2.0.0    
         2.0.30-asserts    
         2.0.29-lto    
         2.0.28-lto    
         2.0.27-lto    
         2.0.26-lto    
         2.0.23-lto    
         2.0.20-lto    
         2.0.19-lto    
         1.40.1    INSTALLED
         1.40.0    
         1.39.20    
         1.39.19    
         1.39.18    
         1.39.17    
         1.39.16    
         1.39.15    
         1.39.14    
         1.39.13    
         1.39.12    
         1.39.11    
         1.39.10    
         1.39.9    
         1.39.8    
         1.39.7    
         1.39.6    
         1.39.5    
         1.39.4    
         1.39.3    
         1.39.2    
         1.39.1    
         1.39.0    
         1.38.48    
         1.38.47    
         1.38.46    
         1.38.45    
         1.38.44    
         1.38.43    
         1.38.42    
         1.38.41    
         1.38.40    
         1.38.39    
         1.38.38    
         1.38.37    
         1.38.36    
         1.38.35    
         1.38.34    
         1.38.33    

The additional following precompiled SDKs are also available for download:
         sdk-releases-upstream-e13a2d74c5fa5f175ae7cffd4197fe7f78bea304-64bit
         sdk-releases-upstream-c69458f1bbf3ef5b8da4e934de210659cc9bca04-64bit
         sdk-releases-fastcomp-edf24e7233e0def312a08cc8dcec63a461155da1-64bit
         sdk-releases-fastcomp-536568644fd67d53778f6111fdd5f64ad3f4c539-64bit
         sdk-fastcomp-1.38.30-64bit
         sdk-fastcomp-1.38.31-64bit

The following SDKs can be compiled from source:
         sdk-upstream-main-64bit  
         sdk-fastcomp-tag-1.38.30-64bit
         sdk-fastcomp-tag-1.38.31-64bit

The following precompiled tool packages are available for download:
           releases-upstream-e13a2d74c5fa5f175ae7cffd4197fe7f78bea304-64bit
           releases-upstream-c69458f1bbf3ef5b8da4e934de210659cc9bca04-64bit
           releases-fastcomp-edf24e7233e0def312a08cc8dcec63a461155da1-64bit
           releases-fastcomp-536568644fd67d53778f6111fdd5f64ad3f4c539-64bit
           fastcomp-clang-e1.38.30-64bit
           fastcomp-clang-e1.38.31-64bit
    (*)    node-14.15.5-64bit           INSTALLED
     *     python-3.9.2-1-64bit         INSTALLED
           emscripten-1.38.30       
           emscripten-1.38.31       

The following tools can be compiled from source:
           llvm-git-main-32bit      
           llvm-git-main-64bit      
           clang-tag-e1.38.30-32bit 
           clang-tag-e1.38.31-32bit 
           fastcomp-clang-tag-e1.38.30-64bit
           fastcomp-clang-tag-e1.38.31-64bit
           fastcomp-clang-master-32bit
           fastcomp-clang-master-64bit
           emscripten-tag-1.38.30-32bit
           emscripten-tag-1.38.31-32bit
           emscripten-tag-1.38.30-64bit
           emscripten-tag-1.38.31-64bit
           binaryen-tag-1.38.30-32bit
           binaryen-tag-1.38.31-32bit
           binaryen-tag-1.38.30-64bit
           binaryen-tag-1.38.31-64bit
           emscripten-main-32bit    
           emscripten-main-64bit    
           binaryen-main-32bit      
           binaryen-main-64bit      
           ninja-git-release-64bit  
           ccache-git-emscripten-64bit

Items marked with * are activated for the current user.
Items marked with (*) are selected for use, but your current shell environment is not configured to use them. Type "source ./emsdk_env.sh" to set up your current shell to use them.

To access the historical archived versions, type 'emsdk list --old'

Run "git pull" to pull in the latest list.

and

$ ./emcc -v
emcc -v
cache:INFO: generating system asset: is_vanilla.txt... (this will be cached in "/Users/richard/projects/emscripten-core/emsdk/upstream/emscripten/cache/is_vanilla.txt" for subsequent builds)
cache:INFO:  - ok
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 1.40.1
clang version 12.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-llvm-llvm--project 9f21947a331203ee2579db87f1d1ec22a949e20a)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Users/richard/projects/emscripten-core/emsdk/upstream/bin
shared:INFO: (Emscripten: Running sanity checks)

Should I do extra jobs to make it work?

Thx for your reading here :P

[richardo2016]

Well, after reading some history commits, I found maybe ./lib/lexer.js was written maually, rather than generated by emcc (written in makfile)?

I will appreciate for your reply. @guybedford

[guybedford]

@richardo2016 emcc will generate lib/lexer.js per that command, but getting to the final result requires manual manipulation yes.

[richardo2016]

@richardo2016 emcc will generate lib/lexer.js per that command, but getting to the final result requires manual manipulation yes.

Thx for your reply!

Sure enough, I found the parts of source in lib/lexer.js is manual-like, and the other parts is generated-like.

I try to resolve this issue: #67 . It seems to be required to regenerate lib/lexer.js if I supported string imports/exports. So if I finished it and make PR to repo. Could you do the regeneration?

[guybedford]

Of course, would be happy to take a PR and manually do the build. There was actually a similar PR to cjs-module-lexer recently here - https://github.com/guybedford/cjs-module-lexer/pull/54. Perhaps compare your results on that, also noting the pure JS implementation as well, which can be nice to keep in sync.

[guybedford]

Oh sorry of course that PR is not related code, slow on context switching today apparently.

[richardo2016]

Of course, would be happy to take a PR and manually do the build. There was actually a similar PR to cjs-module-lexer recently here - guybedford/cjs-module-lexer#54. Perhaps compare your results on that, also noting the pure JS implementation as well, which can be nice to keep in sync.

No problem.

Oh sorry of course that PR is not related code, slow on context switching today apparently.

That's OK, I have read this comment when I want to resolve this issue #67 . Maybe I should do little normalization to track changes in guybedford/cjs-module-lexer.

[guybedford]

It would be a huge help, thank you so much for looking into this.

[richardo2016]

It seems I should use emcc 1.40.1-fastcomp (legacy engine) rather emcc 1.40.1. After changing it, I can generate lib/lexer.asm.js.

But I haven't compiled a valid result, on top of this fix commit, I run commands below to compile lib/lexer.asm.js success:

rm -f lib/lexer.asm.js && make lib/lexer.asm.js

But I don't know how to run tests, commands below seems wrong:

npm run build:asm && npm run test:js

I got error:

So I took a diff view to compare valid lib/lexer.asm.js with local one I compiled:

Should I follow the instructions at the start of left view(the VALID revision) to manipulate codes in right view(the revision I compiled)

[richardo2016]

@guybedford Hi, another PR so soon again :P , but it's not about this issue. After this commit, I can generate lib/lexer.js as expected, just some correction to Makefile required as in PR before. This issue could be closed.