msys2 / MINGW-packages

Package scripts for MinGW-w64 targets to build under MSYS2.
https://packages.msys2.org
BSD 3-Clause "New" or "Revised" License
2.22k stars 1.19k forks source link

node rollup does not work #20407

Open Kreijstal opened 4 months ago

Kreijstal commented 4 months ago

Description / Steps to reproduce the issue

require node rollup use rollup -c will give you an esoteric error, but it is clear that it was made with msvc in mind, we can see this, because when we npm install rollup we get: https://www.npmjs.com/package/@rollup/rollup-win32-x64-msvc , Not sure if either npm would have to be patched, or rollup would be just patched and installed as a msys2 package.. (If that can be figured out). Anyway, also reporting this for future viewers. And people that happen to come across this https://github.com/tauri-apps/tauri/issues/6685

Expected behavior

gives a meaningful error or it works

Actual behavior

$ npm run build

> js-interpreter-npm@1.0.8 build
> rollup -c

thread '<unnamed>' panicked at C:\Users\runneradmin\.cargo\registry\src\index.crates.io-6f17d22bba15001f\napi-sys-2.3.0\src\functions.rs:7:3:
Must load N-API bindings
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Verification

Windows Version

MINGW64_NT-10.0-22631 ayylmai 3.4.10.x86_64 2024-02-10 08:39 UTC x86_64 Msys

MINGW environments affected

Are you willing to submit a PR?

if I figure out a solution

Kreijstal commented 4 months ago

Alternatively, it seems webpack has no trouble with it.

raedrizqie commented 4 months ago

it is not about npm

Kreijstal commented 4 months ago

going through the dependency trail now attempting to make it work with napi-rs. https://github.com/napi-rs/napi-rs/issues/2001

Load Node-API [napi_get_last_error_info] from host runtime failed: GetProcAddress failed
Load Node-API [napi_get_uv_event_loop] from host runtime failed: GetProcAddress failed
Load Node-API [napi_fatal_exception] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_threadsafe_function] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_date] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_bigint_int64] from host runtime failed: GetProcAddress failed
Load Node-API [napi_detach_arraybuffer] from host runtime failed: GetProcAddress failed
Load Node-API [napi_add_async_cleanup_hook] from host runtime failed: GetProcAddress failed
Load Node-API [node_api_symbol_for] from host runtime failed: GetProcAddress failed
Load Node-API [node_api_create_external_string_latin1] from host runtime failed: GetProcAddress failed

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src\lib.rs (target\debug\deps\napi-dec594a7a0918046.exe)
Load Node-API [napi_get_last_error_info] from host runtime failed: GetProcAddress failed
Load Node-API [napi_get_uv_event_loop] from host runtime failed: GetProcAddress failed
Load Node-API [napi_fatal_exception] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_threadsafe_function] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_date] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_bigint_int64] from host runtime failed: GetProcAddress failed
Load Node-API [napi_detach_arraybuffer] from host runtime failed: GetProcAddress failed
Load Node-API [napi_add_async_cleanup_hook] from host runtime failed: GetProcAddress failed
Load Node-API [node_api_symbol_for] from host runtime failed: GetProcAddress failed
Load Node-API [node_api_create_external_string_latin1] from host runtime failed: GetProcAddress failed

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src\lib.rs (target\debug\deps\napi_bench-d35302d6888ef6b3.exe)
Load Node-API [napi_get_last_error_info] from host runtime failed: GetProcAddress failed
Load Node-API [napi_get_uv_event_loop] from host runtime failed: GetProcAddress failed
Load Node-API [napi_fatal_exception] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_threadsafe_function] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_date] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_bigint_int64] from host runtime failed: GetProcAddress failed
Load Node-API [napi_detach_arraybuffer] from host runtime failed: GetProcAddress failed
Load Node-API [napi_add_async_cleanup_hook] from host runtime failed: GetProcAddress failed
Load Node-API [node_api_symbol_for] from host runtime failed: GetProcAddress failed
Load Node-API [node_api_create_external_string_latin1] from host runtime failed: GetProcAddress failed

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src\lib.rs (target\debug\deps\napi_build-c42c8f0fb53550c7.exe)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src\lib.rs (target\debug\deps\napi_compat_mode_examples-dee832aaf9cd1cc4.exe)
Load Node-API [napi_get_last_error_info] from host runtime failed: GetProcAddress failed
Load Node-API [napi_get_uv_event_loop] from host runtime failed: GetProcAddress failed
Load Node-API [napi_fatal_exception] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_threadsafe_function] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_date] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_bigint_int64] from host runtime failed: GetProcAddress failed
Load Node-API [napi_detach_arraybuffer] from host runtime failed: GetProcAddress failed
Load Node-API [napi_add_async_cleanup_hook] from host runtime failed: GetProcAddress failed
Load Node-API [node_api_symbol_for] from host runtime failed: GetProcAddress failed
Load Node-API [node_api_create_external_string_latin1] from host runtime failed: GetProcAddress failed

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src\lib.rs (target\debug\deps\napi_derive-44dc2927c7b568c9.exe)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src\lib.rs (target\debug\deps\napi_derive_backend-2cb26fecfc5f6046.exe)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src\lib.rs (target\debug\deps\napi_examples-9d28c32e18bf0c7e.exe)
Load Node-API [napi_get_last_error_info] from host runtime failed: GetProcAddress failed
Load Node-API [napi_get_uv_event_loop] from host runtime failed: GetProcAddress failed
Load Node-API [napi_fatal_exception] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_threadsafe_function] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_date] from host runtime failed: GetProcAddress failed
Load Node-API [napi_create_bigint_int64] from host runtime failed: GetProcAddress failed
Load Node-API [napi_detach_arraybuffer] from host runtime failed: GetProcAddress failed
Load Node-API [napi_add_async_cleanup_hook] from host runtime failed: GetProcAddress failed
Load Node-API [node_api_symbol_for] from host runtime failed: GetProcAddress failed
Load Node-API [node_api_create_external_string_latin1] from host runtime failed: GetProcAddress failed

So, it builds fine, it gives you a hello world, but not sure what is exactly wrong here.

raedrizqie commented 4 months ago

because it donwloads the prebuilt msvc-based rollup modules rollup.win32-x64-msvc.node:

npm http fetch GET 200 https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.13.0.tgz 1634ms (cache miss)

we need to build rollup.win32-x64-msvc.node from source

Kreijstal commented 4 months ago

because it donwloads the prebuilt msvc-based rollup modules rollup.win32-x64-msvc.node:

npm http fetch GET 200 https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.13.0.tgz 1634ms (cache miss)

we need to build rollup.win32-x64-msvc.node from source

yeah, but for that we need to fix upstream dependencies, like napi-rs

Kreijstal commented 4 months ago

just created this small check to check if nodejs can't be loaded but it can..

#include <stdio.h>
#include <windows.h>
#include <string.h>
#include <node/node_api.h>
int main() {
    // Print the MSYSTEM_PREFIX environment variable
    char* msystemPrefix = getenv("MSYSTEM_PREFIX");
    if (msystemPrefix == NULL) {
        printf("MSYSTEM_PREFIX environment variable is not set\n");
        return 1;
    }
    printf("MSYSTEM_PREFIX = %s\n", msystemPrefix);

    // Construct the path to libnode.dll
    char libNodePath[MAX_PATH];
    snprintf(libNodePath, sizeof(libNodePath), "%s\\bin\\libnode.dll", msystemPrefix);

    // Load the DLL
    HMODULE libNode = LoadLibrary(libNodePath);
    if (libNode == NULL) {
        printf("Failed to load %s\n", libNodePath);
        printf("GetLastError() = %lu\n", GetLastError());
        return 1;
    }
    printf("Successfully loaded %s\n", libNodePath);

    // Get the function pointer
    FARPROC napi_get_last_error_info = GetProcAddress(libNode, "napi_get_last_error_info");
    if (napi_get_last_error_info == NULL) {
        printf("Failed to get address of napi_get_last_error_info\n");
        printf("GetLastError() = %lu\n", GetLastError());
        FreeLibrary(libNode);
        return 1;
    }
    printf("Successfully retrieved address of napi_get_last_error_info\n");

    // Write success message to stdout
    printf("Successfully retrieved address of napi_get_last_error_info\n");

    // Free the library
    BOOL freeResult = FreeLibrary(libNode);
    if (!freeResult) {
        printf("Failed to free %s\n", libNodePath);
        printf("GetLastError() = %lu\n", GetLastError());
    } else {
        printf("Successfully freed %s\n", libNodePath);
    }

    return 0;
}

returns

MSYSTEM_PREFIX = C:/Users/topkek/scoop/apps/msys2/2024-01-13/ucrt64
Successfully loaded C:/Users/topkek/scoop/apps/msys2/2024-01-13/ucrt64\bin\libnode.dll
Successfully retrieved address of napi_get_last_error_info
Successfully retrieved address of napi_get_last_error_info
Successfully freed C:/Users/topkek/scoop/apps/msys2/2024-01-13/ucrt64\bin\libnode.dll
Kreijstal commented 3 months ago

I managed to reduce the bug surface a bit in case anyone is good with rust

#!/bin/bash

# Clone the napi-rs repository with a depth of 1
git clone --depth 1 https://github.com/napi-rs/napi-rs.git

# Create a sibling directory for the Cargo project
mkdir -p sibling/src

# Change to the sibling directory
cd sibling

# Update the Cargo.toml file to include napi-rs as a dependency
echo '[package]
name = "sibling"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]

[dependencies]
napi = { path = "../napi-rs/crates/napi", default-features = false, features = ["napi4"] }' > Cargo.toml

# Create a minimal src/lib.rs file
echo 'use napi::bindgen_prelude::*;' > src/lib.rs

# Build the project
cargo build

# Copy the generated DLL and rename it with a .node extension
cp target/debug/sibling.dll sibling.node

# Run the Node.js module
node sibling.node

The problem is libloading, it doesn't seem to want to load the functions but I'm not 100% sure either how Libloading is supposed to work, in case anyone is curious about how this magic works, Bindgen prelude exposes a dll function napi_register_module_v1 that node calls while registering the module, however some code is executed before it using the [ctor] attribute from the ctor create, all functions from this crate execute when the dll is loaded alledgedly

Kreijstal commented 3 months ago

I am puzzled trying to understand how the rust version works, it attempts to cast GetProcAddress so here I try to recreate it.

#!/bin/bash

# Create the hello.c file
cat > hello.c << EOL
#include <assert.h>
#include <node/node_api.h>

napi_value Method(napi_env env, napi_callback_info info) {
    napi_status status;
    napi_value world;
    status = napi_create_string_utf8(env, "world", NAPI_AUTO_LENGTH, &world);
    assert(status == napi_ok);
    return world;
}

#define DECLARE_NAPI_METHOD(name, func)                                        \\
    { name, 0, func, 0, 0, 0, napi_default, 0 }

napi_value napi_register_module_v1(napi_env env, napi_value exports) {
    napi_status status;
    napi_property_descriptor desc = DECLARE_NAPI_METHOD("hello", Method);
    status = napi_define_properties(env, exports, 1, &desc);
    assert(status == napi_ok);
    return exports;
}
EOL

# Compile the hello.c file
gcc -shared -fPIC hello.c -lnode -o hello.node

# Run the compiled module with Node.js
echo "console.log(require('./hello.node').hello())" | node -

this works so then I try

gendef $MSYSTEM_PREFIX/bin/libnode.dll
dlltool -d libnode.def -l libnode.lib
gcc -shared -fPIC hello.c libnode.lib -o hello.node

apparently this does runtime linking but not in the way I wanted, this still works, but does not help me understand why they try to call GetProcAddress

Kreijstal commented 3 months ago

so I recreated the toy example using GetProcAddress, and in mingw it still works, what I do not see is how they are loading the dll.

#include <assert.h>
#include <node/node_api.h>
#include <windows.h>
#include <stdio.h>

HMODULE nodeModule = NULL;
typedef napi_status (*napi_create_string_utf8_func)(napi_env env,
                                                    const char* str,
                                                    size_t length,
                                                    napi_value* result);
typedef napi_status (*napi_define_properties_func)(napi_env env,
                                                   napi_value object,
                                                   size_t property_count,
                                                   const napi_property_descriptor* properties);
napi_create_string_utf8_func napi_create_string_utf8dyn =NULL;
napi_define_properties_func napi_define_propertiesdyn =NULL;
bool LoadNodeFunctions() {
    nodeModule = LoadLibraryA("libnode.dll");
    if (!nodeModule) {
        printf("Failed to load Node.js DLL\n");
        return false;

   }
     napi_create_string_utf8dyn = (napi_create_string_utf8_func)GetProcAddress(nodeModule, "napi_create_string_utf8");
     napi_define_propertiesdyn = (napi_define_properties_func)GetProcAddress(nodeModule, "napi_define_properties");

    if (!napi_create_string_utf8 || !napi_define_properties) {
        printf("Failed to get function addresses\n");
        FreeLibrary(nodeModule);
        return false;
    }

    return true;
}

napi_value Method(napi_env env, napi_callback_info info) {
    napi_status status;
    napi_value world;
    status = napi_create_string_utf8dyn(env, "world", NAPI_AUTO_LENGTH, &world);
    assert(status == napi_ok);

    return world;
}

#define DECLARE_NAPI_METHOD(name, func)                                        \
    { name, 0, func, 0, 0, 0, napi_default, 0 }

napi_value napi_register_module_v1(napi_env env, napi_value exports) {
    if (!nodeModule) {
        if (!LoadNodeFunctions()) {
            return exports;
        }
    }
    napi_status status;
    napi_property_descriptor desc = DECLARE_NAPI_METHOD("hello", Method);
    status = napi_define_propertiesdyn(env, exports, 1, &desc);
    assert(status == napi_ok);

    return exports;
}

This will compile without linking to libnode.dll explizitly, but it links on runtime, (I imagine they do that so it works on many nodejs versions)

Kreijstal commented 3 months ago

someone might have a similar issue here https://github.com/nodejs/node-gyp/pull/2834

Kreijstal commented 3 months ago

and I manage to reproduce the mistake of napi-rs

#include <assert.h>
#include <node/node_api.h>
#include <windows.h>
#include <stdio.h>
#include <tlhelp32.h>

#include <psapi.h>

void printMyProcess() {
    // Get the current process ID
    DWORD processId = GetCurrentProcessId();

    // Get a handle to the current process
    HANDLE hProcess = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, processId);
    if (hProcess == NULL) {
        printf("Failed to open process\n");
        return;
    }

    // Get the process name
    char processName[MAX_PATH];
    if (!GetModuleFileNameEx(hProcess, NULL, processName, MAX_PATH)) {
        printf("Failed to get process name\n");
        CloseHandle(hProcess);
        return;
    }

    // Print the process name
    printf("Current process name: %s\n", processName);

    // Close the handle to the process
    CloseHandle(hProcess);
}
typedef enum {
    GET_MODULE_HANDLE_EX_W_UNKNOWN,
    GET_MODULE_HANDLE_EX_W_ERROR
} GetModuleHandleExWError;

GetModuleHandleExWError GetModuleHandleExWErrorFromLastError() {
    DWORD lastError = GetLastError();
    if (lastError != 0) {
        return GET_MODULE_HANDLE_EX_W_ERROR;
    }
    return GET_MODULE_HANDLE_EX_W_UNKNOWN;
}

HMODULE GetCurrentModuleHandle() {
    HMODULE handle = NULL;
    BOOL result = GetModuleHandleExW(0,NULL,
        &handle
    );

    if (result == 0) {
        GetModuleHandleExWError error = GetModuleHandleExWErrorFromLastError();
        if (error == GET_MODULE_HANDLE_EX_W_ERROR) {
            printf("GetModuleHandleExW error: %lu\n", GetLastError());
        } else {
            printf("Unknown GetModuleHandleExW error\n");
        }
        return NULL;
    }

    return handle;
}
HMODULE nodeModule = NULL;
typedef napi_status (*napi_create_string_utf8_func)(napi_env env,
                                                    const char* str,
                                                    size_t length,
                                                    napi_value* result);
typedef napi_status (*napi_define_properties_func)(napi_env env,
                                                   napi_value object,
                                                   size_t property_count,
                                                   const napi_property_descriptor* properties);
napi_create_string_utf8_func napi_create_string_utf8dyn =NULL;
napi_define_properties_func napi_define_propertiesdyn =NULL;
bool LoadNodeFunctions() {
#ifdef LOAD_FROM_PROCESS
    nodeModule = GetCurrentModuleHandle();
#else
    nodeModule = LoadLibraryA("libnode.dll");
#endif
    if (!nodeModule) {
        printf("Failed to load Node.js DLL\n");
        return false;
    }

    napi_create_string_utf8dyn = (napi_create_string_utf8_func)GetProcAddress(nodeModule, "napi_create_string_utf8");
    napi_define_propertiesdyn = (napi_define_properties_func)GetProcAddress(nodeModule, "napi_define_properties");

    if (!napi_create_string_utf8dyn || !napi_define_propertiesdyn) {
        printf("Failed to get function addresses\n");
#ifndef LOAD_FROM_PROCESS
        FreeLibrary(nodeModule);
#endif
        return false;
    }

    return true;
}

napi_value Method(napi_env env, napi_callback_info info) {
    napi_status status;
    napi_value world;
    status = napi_create_string_utf8dyn(env, "world", NAPI_AUTO_LENGTH, &world);
    assert(status == napi_ok);

    return world;
}

#define DECLARE_NAPI_METHOD(name, func)                                        \
    { name, 0, func, 0, 0, 0, napi_default, 0 }

napi_value napi_register_module_v1(napi_env env, napi_value exports) {
    printMyProcess();
    if (!nodeModule) {
        if (!LoadNodeFunctions()) {
            return exports;
        }
    }
    napi_status status;
    napi_property_descriptor desc = DECLARE_NAPI_METHOD("hello", Method);
    status = napi_define_propertiesdyn(env, exports, 1, &desc);
    assert(status == napi_ok);

    return exports;
}

compile as gcc -DLOAD_FROM_PROCESS -shared -fPIC hello.c -o hello.node and you get Failed to get function addresses which makes sense

Kreijstal commented 3 months ago

Just noticed that other people solve this problem by just statically linking

https://users.rust-lang.org/t/why-is-msvc-the-default-toolchain/44287/7

I guess we can do that, it is much faster as well..

Kreijstal commented 3 months ago

patching upstream https://github.com/napi-rs/napi-rs/pull/2026

Kreijstal commented 1 month ago

napi-rs has finally been merged, but that's half the battle, now you need to submit pull requests to all napi-rs dependents to use the new napi-rs version

raedrizqie commented 1 month ago

that is still not enough.. every napi-rs based package must update their support for windows gnu variant.. else, the msvc-based binary would get downloaded and used..

Kreijstal commented 1 month ago

Since you are submitting pull request to all depends then you just add the option of using the gnu napi... No?

raedrizqie commented 1 month ago

napi now works for us, but not for other packages.. npm just check for windows and bitness.. thats all.

thats why we still can build those packages, but with msvc binary.. which will fail at runtime.. these only affect binary packages. pure nodejs packages are not affected.. its the same story as python pip..

benzidane027 commented 1 month ago

same error . any solution ??

Kreijstal commented 1 month ago

same error . any solution ??

well, the new version of napi-rs should support gnu, but you need to update the dependencies so that this is the case.