Missing lua_open() on lua 5.3

[marandus]

When using lua 5.3, the installation of this tool fails because lua_open() is missing. I found this post stating that lua_open() was removed in 5.3. http://lua-users.org/lists/lua-l/2006-07/msg00329.html

[CodeLenny]

Hi @marandus. I'm sorry you're hitting an error, I'm looking into this now.

It looks like I was using this tool under lua5.1 (as specified in .travis.yml).

I'm not sure where you're hitting lua_open() - I can't find anywhere I use lua_open() in my code, but it's possible I'm missing something. If you've got any stack traces, I'd love to look into this further.

If it's easy to fix factorio-recipe-extraction to work with lua5.3, I'll happily implement the fix (and update Travis CI to ensure that 5.3 is also being tested!). Otherwise I'll update the README to document this library as only working under 5.1.

[marandus]

It seems the issue is more with nodelua instead of your code. After posting this issue I found out that it was actually nodelua that did not build properly. Sorry for that.

I installed lua 5.1 next to the already existing 5.2 and 5.3, and then the nodelua build finished. But now I cannot start your tool, I get an error about a missing symbol lua_pcall, which should be part of lua's core library. It seems that the nodelua version you are using does not work well on my setup and loads the wrong shared object. I don't know if there is a newer version available that will fix this behavior.

[CodeLenny]

Ah, right, thank you very much for digging a little more. Now I remember the problems with NodeLua! I'll write the requirement of 5.1 into the README (#10).

The last build I ran used NodeLua from the huuugegames/NodeLua fork (Travis Log, which hasn't been updated since January 11th, so I don't think you've got a different version of NodeLua.

Mind giving me a few more clues?

• What OS are you using? (+ OS Version)
• How did you install Lua 5.1? (Tests are using the lua5.1-dev package from Ubuntu)

If you could copy/paste any logs containing the error, I might be able to spot something.

Again, thanks for doing some more digging!

[CodeLenny]

Also FYI, I just re-ran all the tests, and everything passed (https://travis-ci.org/CodeLenny/factorio-recipe-extraction/builds/323005498), so I haven't been able to reproduce your error. I'll be able to help more with the extra details and/or the log.

[marandus]

I am on Antergos (based on Arch, rolling release) and installed the lua51 package using pacman. The default lua package installs the newest version, which is 5.3, and the lua52 is installed as well. lua and lua52 were already installed, I just added lua51 into the mix.

All executables live alongside each other in /usr/bin/ with version suffixes. The default executables without suffix are the same as the 5.3 ones.

• lua5.1, luac5.1 for 5.1
• lua5.2, luac5.2 for 5.2
• lua, luac, lua5.3, luac5.3 for 5.3

Version output from these executables:

$ lua5.1 -v
Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio

$ lua5.2 -v
Lua 5.2.4  Copyright (C) 1994-2015 Lua.org, PUC-Rio

$ lua5.3 -v
Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio

$ lua -v
Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio

I installed your application using npm install factorio-recipe-extraction. During installation it tried to built nodelua, which failed until I installed the lua51 package. Afterwards the build succeeded.

From the install directory I call the following:

$(npm bin)/factorio-extractor --output test.json --data ~/.factorio

which leads to this error:

node: symbol lookup error: /home/username/Applications/factorio-recipe-extraction/node_modules/nodelua/build/Release/nodelua.node: undefined symbol: lua_pcall

My best guess would be that for some reason nodelua loads the lua 5.3 binaries instead of the 5.1 ones it was compiled with and thus fails to resolve the symbol reference to lua_pcall. But since I have no idea how nodelua works this is just a guess.

[marandus]

Additional info, output of npm install:

$ npm install factorio-recipe-extraction

> nodelua@0.4.1 install /home/username/Applications/factorio-recipe-extraction/node_modules/nodelua
> node-gyp rebuild

make: Verzeichnis „/home/username/Applications/factorio-recipe-extraction/node_modules/nodelua/build“ wird betreten
  CXX(target) Release/obj.target/nodelua/src/utils.o
  CXX(target) Release/obj.target/nodelua/src/luastate.o
../src/luastate.cc: In statischer Elementfunktion »static void LuaState::New(const v8::FunctionCallbackInfo<v8::Value>&)«:
../src/luastate.cc:217:21: Warnung: logisches Nicht bezieht sich nur auf die linke Seite des Vergleichs [-Wlogical-not-parentheses]
   if(!args.Length() > 0){
                     ^
../src/luastate.cc:217:6: Anmerkung: klammern Sie den Ausdruck auf der linken Seite ein, um diese Warnung zu beheben
   if(!args.Length() > 0){
      ^~~~~~~~~~~~~~
      (             )
  CXX(target) Release/obj.target/nodelua/src/nodelua.o
  SOLINK_MODULE(target) Release/obj.target/nodelua.node
  COPY Release/nodelua.node
make: Verzeichnis „/home/username/Applications/factorio-recipe-extraction/node_modules/nodelua/build“ wird verlassen
npm WARN saveError ENOENT: no such file or directory, open '/home/username/Applications/factorio-recipe-extraction/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/home/username/Applications/factorio-recipe-extraction/package.json'
npm WARN factorio-recipe-extraction No description
npm WARN factorio-recipe-extraction No repository field.
npm WARN factorio-recipe-extraction No README data
npm WARN factorio-recipe-extraction No license field.

+ factorio-recipe-extraction@0.1.1
added 61 packages in 6.686s

[CodeLenny]

Hmm. I don't get as many errors during npm install (actually, 0 warnings/errors): https://travis-ci.org/CodeLenny/factorio-recipe-extraction/builds/323005498#L599

Looks like nodelua only has a single use of lua_pcall: https://github.com/HuuugeGames/NodeLua/blob/master/src/utils.cc#L42

I'd like to see if the unit tests are passing on your machine, especially from a clean start. Would you mind running the following?

cd ~/
git clone https://github.com/CodeLenny/factorio-recipe-extraction.git tmp-factorio-recipe-extraction
cd tmp-factorio-recipe-extraction
npm install
npm test

(Clones the source repository into a new directory to get a fresh start, then does npm install to get a clean copy of all deps, then npm test to run the unit tests. Shouldn't make any changes to the filesystem outside of the newly created directory)

[marandus]

Did the above, output follows. Just for reference: I have GCC 7.2.1 installed.

gcc-Version 7.2.1 20171128 (GCC)

If I find time, I will do some further testing with a clean Arch Linux VM that has only lua 5.1 installed. Just to see if the installation of multiple lua versions side by side breaks nodelua. But because of new year's eve and stuff this could take some time.

But now: output! First npm install:

$ npm install

> nodelua@0.4.1 install /home/username/Sources/tmp-factorio-recipe-extraction/node_modules/nodelua
> node-gyp rebuild

make: Verzeichnis „/home/username/Sources/tmp-factorio-recipe-extraction/node_modules/nodelua/build“ wird betreten
  CXX(target) Release/obj.target/nodelua/src/utils.o
  CXX(target) Release/obj.target/nodelua/src/luastate.o
../src/luastate.cc: In statischer Elementfunktion »static void LuaState::New(const v8::FunctionCallbackInfo<v8::Value>&)«:
../src/luastate.cc:217:21: Warnung: logisches Nicht bezieht sich nur auf die linke Seite des Vergleichs [-Wlogical-not-parentheses]
   if(!args.Length() > 0){
                     ^
../src/luastate.cc:217:6: Anmerkung: klammern Sie den Ausdruck auf der linken Seite ein, um diese Warnung zu beheben
   if(!args.Length() > 0){
      ^~~~~~~~~~~~~~
      (             )
  CXX(target) Release/obj.target/nodelua/src/nodelua.o
  SOLINK_MODULE(target) Release/obj.target/nodelua.node
  COPY Release/nodelua.node
make: Verzeichnis „/home/username/Sources/tmp-factorio-recipe-extraction/node_modules/nodelua/build“ wird verlassen
npm notice created a lockfile as package-lock.json. You should commit this file.
added 549 packages in 14.695s

Second: npm test:

$ npm test

> factorio-recipe-extraction@0.1.1 test /home/username/Sources/tmp-factorio-recipe-extraction
> npm-run-all lint unit-test

> factorio-recipe-extraction@0.1.1 lint /home/username/Sources/tmp-factorio-recipe-extraction
> eslint .

> factorio-recipe-extraction@0.1.1 unit-test /home/username/Sources/tmp-factorio-recipe-extraction
> nyc --reporter=html --reporter=text mocha

(node:4137) Warning: process.on(SIGPROF) is reserved while debugging

  [integration] Extractor#extract
    1) should have 1 versions of Factorio for testing

  Mod#dependencies
    ✓ returns the dependencies from the manifest
    ✓ includes 'core' when name === 'base'

  Mod#dependsOn
    ✓ should find mods in the dependencies
    ✓ should include optional mods by default
    ✓ should return 'false' if the mod isn't in the dependencies
    'ignoreOptional' === true
      ✓ should include non-optional mods
      ✓ should exclude optional mods

  Mod.loadFromManifest
    2) should have 1 versions of Factorio for testing

  Mod.loadFromZip
    ✓ loads (94ms)

  Mod#satisfiesDependency
    ✓ should reject dependencies with different names
    ✓ should accept dependencies with null versions
    DependencyType.EqualTo
      ✓ should reject higher versions
      ✓ should reject lower versions
      ✓ should accept equal versions
    DependencyType.GreaterThan
      ✓ should accept higher versions
      ✓ should reject lower versions
      ✓ should reject equal versions
    DependencyType.GreaterThanOrEqual
      ✓ should accept higher versions
      ✓ should reject lower versions
      ✓ should accept equal versions

  Mod#version
    ✓ gets the version from the manifest
    ✓ defaults to '0.0.0.0'

  Mod.versionGreater
    returns 'true' when version is larger
      ✓ versionGreater(0.1.1.10, 0.1.1.5) === true
      ✓ versionGreater(1.0.0.0, 0.0.0.0) === true
      ✓ versionGreater(0.10.0.0, 0.5.0.0) === true
      ✓ versionGreater(0.10.0.5, 0.5.0.10) === true
      ✓ versionGreater(10.0.0.5, 5.0.0.10) === true
      ✓ versionGreater(10.3.2.1, 5.1.2.3) === true
      ✓ versionGreater(10, 5) === true
      ✓ versionGreater(10.0.0.0, 5) === true
      ✓ versionGreater(0.10.5, 0.5) === true
    returns 'false' when versions are equal
      ✓ versionGreater(0.0.0.0, 0.0.0.0) === false
      ✓ versionGreater(1.1.1.1, 1.1.1.1) === false
      ✓ versionGreater(0.1.2.3, 0.1.2.3) === false
      ✓ versionGreater(5, 5) === false
    returns 'false' when version is smaller
      ✓ versionGreater(0.1.1.5, 0.1.1.10) === false
      ✓ versionGreater(0.0.0.0, 1.0.0.0) === false
      ✓ versionGreater(0.5.0.0, 0.10.0.0) === false
      ✓ versionGreater(0.5.0.10, 0.10.0.5) === false
      ✓ versionGreater(5.0.0.10, 10.0.0.5) === false
      ✓ versionGreater(5.1.2.3, 10.3.2.1) === false
      ✓ versionGreater(5, 10) === false
      ✓ versionGreater(5, 10.0.0.0) === false
      ✓ versionGreater(0.5, 0.10.5) === false

  ModuleLoader#added
    ✓ finds mods

  ModuleLoader#all
    3) should have 1 versions of Factorio for testing

  ModuleLoader#enabled
    given a ModArray
      ✓ includes enabled mods
      ✓ excludes mods listed with 'enabled: false'
      ✓ excludes mods not listed in 'mod-list.json'
    looking up mods
      ✓ should use options in the constructor
      ✓ should accept options directly

  ModuleLoader#sortedDependencies
    ✓ shouldn't change an empty array
    ✓ shouldn't change independent mods
    ✓ should work with parallel dependency trees
    ✓ should work with broad trees
    should list independents before mods with dependencies
      ✓ sortedDependencies([independent, dependent])
      ✓ sortedDependencies([dependent, independent])
      ✓ sortedDependencies([independent, dependent, deeplyDependent])
      ✓ sortedDependencies([independent, deeplyDependent, dependent])
      ✓ sortedDependencies([dependent, independent, deeplyDependent])
      ✓ sortedDependencies([dependent, deeplyDependent, independent])
      ✓ sortedDependencies([deeplyDependent, independent, dependent])
      ✓ sortedDependencies([deeplyDependent, dependent, independent])

  ModuleLoader#vanilla
    4) should have 1 versions of Factorio for testing

  61 passing (6s)
  4 failing

  1) [integration] Extractor#extract should have 1 versions of Factorio for testing:
     AssertionError: Should have at least 1 versions, but found 0: expected 0 to be at least 1
      at Context.<anonymous> (test/assets/FactorioData.js:35:48)

  2) Mod.loadFromManifest should have 1 versions of Factorio for testing:
     AssertionError: Should have at least 1 versions, but found 0: expected 0 to be at least 1
      at Context.<anonymous> (test/assets/FactorioData.js:35:48)

  3) ModuleLoader#all should have 1 versions of Factorio for testing:
     AssertionError: Should have at least 1 versions, but found 0: expected 0 to be at least 1
      at Context.<anonymous> (test/assets/FactorioData.js:35:48)
      at tryCatcher (node_modules/bluebird/js/release/util.js:16:23)
      at Promise._settlePromiseFromHandler (node_modules/bluebird/js/release/promise.js:512:31)
      at Promise._settlePromise (node_modules/bluebird/js/release/promise.js:569:18)
      at Promise._settlePromise0 (node_modules/bluebird/js/release/promise.js:614:10)
      at Promise._settlePromises (node_modules/bluebird/js/release/promise.js:693:18)
      at Async._drainQueue (node_modules/bluebird/js/release/async.js:133:16)
      at Async._drainQueues (node_modules/bluebird/js/release/async.js:143:10)
      at Immediate.Async.drainQueues [as _onImmediate] (node_modules/bluebird/js/release/async.js:17:14)

  4) ModuleLoader#vanilla should have 1 versions of Factorio for testing:
     AssertionError: Should have at least 1 versions, but found 0: expected 0 to be at least 1
      at Context.<anonymous> (test/assets/FactorioData.js:35:48)

-----------------|----------|----------|----------|----------|----------------|
File             |  % Stmts | % Branch |  % Funcs |  % Lines |Uncovered Lines |
-----------------|----------|----------|----------|----------|----------------|
All files        |    57.95 |    62.86 |    51.22 |    56.29 |                |
 Extractor.js    |     7.59 |        0 |        0 |     8.33 |... 241,242,243 |
 Mod.js          |    96.43 |      100 |    95.24 |    95.92 |          45,46 |
 ModuleLoader.js |    88.33 |    71.43 |       88 |    89.13 |... ,77,173,176 |
-----------------|----------|----------|----------|----------|----------------|
npm ERR! code ELIFECYCLE
npm ERR! errno 4
npm ERR! factorio-recipe-extraction@0.1.1 unit-test: `nyc --reporter=html --reporter=text mocha`
npm ERR! Exit status 4
npm ERR! 
npm ERR! Failed at the factorio-recipe-extraction@0.1.1 unit-test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/username/.npm/_logs/2017-12-30T08_05_03_983Z-debug.log
ERROR: "unit-test" exited with 4.
npm ERR! Test failed.  See above for more details.

Running your tool from this new directory yields the same result as before:

$ node ./bin/factorio-extractor.js --output /tmp/factorio.json --data ~/.factorio
node: symbol lookup error: /home/username/Sources/tmp-factorio-recipe-extraction/node_modules/nodelua/build/Release/nodelua.node: undefined symbol: lua_pcall

[marandus]

And contents of the log file mentioned in the test output:

0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node',
1 verbose cli   '/usr/lib/node_modules/npm/bin/npm-cli.js',
1 verbose cli   'run',
1 verbose cli   'unit-test' ]
2 info using npm@5.6.0
3 info using node@v9.3.0
4 verbose run-script [ 'preunit-test', 'unit-test', 'postunit-test' ]
5 info lifecycle factorio-recipe-extraction@0.1.1~preunit-test: factorio-recipe-extraction@0.1.1
6 info lifecycle factorio-recipe-extraction@0.1.1~unit-test: factorio-recipe-extraction@0.1.1
7 verbose lifecycle factorio-recipe-extraction@0.1.1~unit-test: unsafe-perm in lifecycle true
8 verbose lifecycle factorio-recipe-extraction@0.1.1~unit-test: PATH: /usr/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/home/username/Sources/tmp-factorio-recipe-extraction/node_modules/.bin:/usr/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/home/username/Sources/tmp-factorio-recipe-extraction/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
9 verbose lifecycle factorio-recipe-extraction@0.1.1~unit-test: CWD: /home/username/Sources/tmp-factorio-recipe-extraction
10 silly lifecycle factorio-recipe-extraction@0.1.1~unit-test: Args: [ '-c', 'nyc --reporter=html --reporter=text mocha' ]
11 silly lifecycle factorio-recipe-extraction@0.1.1~unit-test: Returned: code: 4  signal: null
12 info lifecycle factorio-recipe-extraction@0.1.1~unit-test: Failed to exec unit-test script
13 verbose stack Error: factorio-recipe-extraction@0.1.1 unit-test: `nyc --reporter=html --reporter=text mocha`
13 verbose stack Exit status 4
13 verbose stack     at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:285:16)
13 verbose stack     at EventEmitter.emit (events.js:159:13)
13 verbose stack     at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:159:13)
13 verbose stack     at maybeClose (internal/child_process.js:943:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:220:5)
14 verbose pkgid factorio-recipe-extraction@0.1.1
15 verbose cwd /home/username/Sources/tmp-factorio-recipe-extraction
16 verbose Linux 4.14.5-1-vfio-npt
17 verbose argv "/usr/bin/node" "/usr/lib/node_modules/npm/bin/npm-cli.js" "run" "unit-test"
18 verbose node v9.3.0
19 verbose npm  v5.6.0
20 error code ELIFECYCLE
21 error errno 4
22 error factorio-recipe-extraction@0.1.1 unit-test: `nyc --reporter=html --reporter=text mocha`
22 error Exit status 4
23 error Failed at the factorio-recipe-extraction@0.1.1 unit-test script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 4, true ]

[CodeLenny]

Oh right, there's a script that downloads the Factorio game data from factorio.com. Sorry about that!

node test/assets/download-headless-data.js needs to be run before npm test to be able to run all the tests.

I'd be interested if the bare Arch install works. If so, you might just need to set NODELUA_INCLUDE or whatever variables are in the NodeLua docs.

[marandus]

Alright, by reinstalling the application with the following environment variables the lua errors go away. My guess with the wrong lua libraries seemed to be correct.

NODELUA_INCLUDE="/usr/include/lua5.1" NODELUA_FLAGS="-llua5.1" npm install factorio-recipe-extraction

However, now I get another error, which is related to the factorio data/installation.

$ $(npm bin)/factorio-extractor --output /tmp/factorio.json --data ~/.factorio
Unhandled rejection Error: Exception In File /home/username/.factorio/data/core/lualib/dataloader.lua Has Failed:
cannot open /home/username/.factorio/data/core/lualib/dataloader.lua: No such file or directory

    at maybeWrapAsError (/home/username/Applications/factorio-recipe-extraction/node_modules/bluebird/js/release/util.js:61:12)
    at /home/username/Applications/factorio-recipe-extraction/node_modules/bluebird/js/release/nodeback.js:38:50

I installed Factorio via Steam and that directory your tool tries to open does not even exist in ~/.factorio. Do I have to point it to the installation directory? And will it detect the mods, which are installed in ~/.factorio instead of the installation directory?

[CodeLenny]

Great, I'll try to write up the NodeLua info in the README at some point. I've already updated it to mention needing 5.1 specifically.

Thank you so much for looking into the Lua issue!

Now, on to the Steam installation. It's been a while since I've used the Steam installation for Factorio, so I don't actually know where things are stored!

In my installation from factorio.com, I've got the following subdirectories:

Do I understand correctly that the Steam installation has split these files between two different locations?

From a quick scan of my code in factorio-recipe-extractor, it looks like I only need data/, mods/*.zip, and mods/mod-list.json (although it's possible there are some more files being used under the hood).

If you can locate those directories, I could create a "steam mode" of the extractor.

[marandus]

The Steam version uses two different directories, that is correct. The game is installed in your Steam library, wherever that is located. But there it is located in the subdirectory steamapps/common/Factorio. This directory contains the game installation, which includes the bin, config, data directories and some more.

Besides that the game itself uses the the ~/.factorio directory for user-specific data, such as mods, save games, etc. Just as the "normal" installation does. I guess you can even install that to a completely different location and it will still use ~/.factorio for that data, but I have not verified that.

The easiest way to solve this would be to split the command line options into two paths, one for the installation directory and one for the user-specific data directory. So for my case I would call the tool with:

$ $(npm bin)/factorio-extractor --output /tmp/factorio.json --game-install ~/STEAMLIB/steamapps/common/Factorio --user-data ~/.factorio

Of course both directories may be the same (as in your case), but then both options would be the same or you could make the --user-data optional and if not set, it uses the same as --game-install.

BTW: I managed to get your tool running but ran into issues with some of my installed mods (I only have Bob and Angel installed). Some data was missing and the tool printed some errors to the terminal. I will try to debug that a little more and if I cannot solve it, I will open a new issue for that since it is no longer related to these problems.

Thank you for your fast responses to my issues, very appreciated!

[CodeLenny]

Thanks for the additional info, I'll look at implementing --game-install!

A hackish solution for now would be to recreate the layout with symlinks:

$ ls -l ~/.fake-factorio
mods -> ~/.factorio/mods
data -> ~/STEAMLIB/steamapps/common/Factorio/data
config -> ~/STEAMLIB/steamapps/common/Factorio/config

Actually, if you wanted to do that, it would actually make sure that there aren't any other differences in the file layout that I should take into account! But I should be able to implement --game-install anyways.

Please feel free to dig more on your own, but I'm also trying to create tests trying to extract various sets of mods - currently I'm just using a couple of 5Dim's mods. I'd be happy to add a combination of Angel's and Bob's mods if you wanted to give me a list. (if I can legally download them during automated tests).

Also feel free to copy/paste a big log file into a new issue, I might be able to recognize the errors and remember what they are trying to say!

[marandus]

Using symlinks was actually my first idea, but it didn't work. Seems like node does not follow symlinks or at least you code does not. When I copied the folders into a new fake directory it worked besides the already mentioned errors.

[marandus]

Oh BTW, I made some screenshots of the two directories. Should help with deciding where to find what.

Installation directory (in Steam library):

User-specific data directory (~/.factorio):