search

Blizzard / node-rdkafka

Node.js bindings for librdkafka
MIT License
2.12k stars 396 forks source link

Compilation error on Heroku due to “File name too long” #385

Closed Aupajo closed 5 years ago

Aupajo commented 6 years ago

I suspect this is not an issue with node-rdkafka, but with librdkafka (or, possibly, node-gyp), but I'll file it here in the hopes that you might be able to help, or at the very least leave this as a starting point for someone in the future.

-----> Building dependencies
       Installing node modules (yarn.lock)
       yarn install v1.5.1
       [1/4] Resolving packages...
       [2/4] Fetching packages...
       info fsevents@1.1.3: The platform "linux" is incompatible with this module.
       info "fsevents@1.1.3" is an optional dependency and failed compatibility check. Excluding it from installation.
       [3/4] Linking dependencies...
       [4/4] Building fresh packages...
       error An unexpected error occurred: "/tmp/build_fdd9e9cba78f819cac297390a51ef95a/SalesforceCloudServices-rbc-rebate-management-ace3bb7b6184ddfbad9427b727eb332cd26f01f3/node_modules/datapipeline-lib/node_modules/node-rdkafka: Command failed.
       Exit code: 1
       Command: sh
       Arguments: -c node-gyp rebuild
       Directory: /tmp/build_fdd9e9cba78f819cac297390a51ef95a/SalesforceCloudServices-rbc-rebate-management-ace3bb7b6184ddfbad9427b727eb332cd26f01f3/node_modules/datapipeline-lib/node_modules/node-rdkafka
       Output:
       make: Entering directory '/tmp/build_fdd9e9cba78f819cac297390a51ef95a/SalesforceCloudServices-rbc-rebate-management-ace3bb7b6184ddfbad9427b727eb332cd26f01f3/node_modules/datapipeline-lib/node_modules/node-rdkafka/build'
       make: stat: _tmp_build_fdd9e9cba78f819cac297390a51ef95a_SalesforceCloudServices_rbc_rebate_management_ace3bb7b6184ddfbad9427b727eb332cd26f01f3_node_modules_datapipeline_lib_node_modules_node_rdkafka_deps_librdkafka_gyp_librdkafka_target_build_dependencies.intermediate: File name too long
       TOUCH _tmp_build_fdd9e9cba78f819cac297390a51ef95a_SalesforceCloudServices_rbc_rebate_management_ace3bb7b6184ddfbad9427b727eb332cd26f01f3_node_modules_datapipeline_lib_node_modules_node_rdkafka_deps_librdkafka_gyp_librdkafka_target_build_dependencies.intermediate
       touch: cannot touch '_tmp_build_fdd9e9cba78f819cac297390a51ef95a_SalesforceCloudServices_rbc_rebate_management_ace3bb7b6184ddfbad9427b727eb332cd26f01f3_node_modules_datapipeline_lib_node_modules_node_rdkafka_deps_librdkafka_gyp_librdkafka_target_build_dependencies.intermediate': File name too long
       deps/librdkafka.target.mk:16: recipe for target '_tmp_build_fdd9e9cba78f819cac297390a51ef95a_SalesforceCloudServices_rbc_rebate_management_ace3bb7b6184ddfbad9427b727eb332cd26f01f3_node_modules_datapipeline_lib_node_modules_node_rdkafka_deps_librdkafka_gyp_librdkafka_target_build_dependencies.intermediate' failed
       rm _tmp_build_fdd9e9cba78f819cac297390a51ef95a_SalesforceCloudServices_rbc_rebate_management_ace3bb7b6184ddfbad9427b727eb332cd26f01f3_node_modules_datapipeline_lib_node_modules_node_rdkafka_deps_librdkafka_gyp_librdkafka_target_build_dependencies.intermediate
       make: Leaving directory '/tmp/build_fdd9e9cba78f819cac297390a51ef95a/SalesforceCloudServices-rbc-rebate-management-ace3bb7b6184ddfbad9427b727eb332cd26f01f3/node_modules/datapipeline-lib/node_modules/node-rdkafka/build'
       make: *** [_tmp_build_fdd9e9cba78f819cac297390a51ef95a_SalesforceCloudServices_rbc_rebate_management_ace3bb7b6184ddfbad9427b727eb332cd26f01f3_node_modules_datapipeline_lib_node_modules_node_rdkafka_deps_librdkafka_gyp_librdkafka_target_build_dependencies.intermediate] Error 1
       make: unlink: _tmp_build_fdd9e9cba78f819cac297390a51ef95a_SalesforceCloudServices_rbc_rebate_management_ace3bb7b6184ddfbad9427b727eb332cd26f01f3_node_modules_datapipeline_lib_node_modules_node_rdkafka_deps_librdkafka_gyp_librdkafka_target_build_dependencies.intermediate: File name too long
       gyp ERR! build error
       gyp ERR! stack Error: `make` failed with exit code: 2
       gyp ERR! stack     at ChildProcess.onExit (/tmp/build_fdd9e9cba78f819cac297390a51ef95a/SalesforceCloudServices-rbc-rebate-management-ace3bb7b6184ddfbad9427b727eb332cd26f01f3/.heroku/node/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:258:23)
       gyp ERR! stack     at emitTwo (events.js:126:13)
       gyp ERR! stack     at ChildProcess.emit (events.js:214:7)
       gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
       gyp ERR! System Linux 4.4.0-1012-aws
       gyp ERR! command "/tmp/build_fdd9e9cba78f819cac297390a51ef95a/SalesforceCloudServices-rbc-rebate-management-ace3bb7b6184ddfbad9427b727eb332cd26f01f3/.heroku/node/bin/node" "/tmp/build_fdd9e9cba78f819cac297390a51ef95a/SalesforceCloudServices-rbc-rebate-management-ace3bb7b6184ddfbad9427b727eb332cd26f01f3/.heroku/node/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
       gyp ERR! cwd /tmp/build_fdd9e9cba78f819cac297390a51ef95a/SalesforceCloudServices-rbc-rebate-management-ace3bb7b6184ddfbad9427b727eb332cd26f01f3/node_modules/datapipeline-lib/node_modules/node-rdkafka
       gyp ERR! node -v v8.10.0
       gyp ERR! node-gyp -v v3.6.2
       gyp ERR! not ok".

Essentially, this fails because during the make build, the filename of the current directory is incorporated into an artefact. In this case, we're running make inside the very long-winded /tmp/build_fdd9e9cba78f819cac297390a51ef95a/SalesforceCloudServices-rbc-rebate-management-ace3bb7b6184ddfbad9427b727eb332cd26f01f3/node_modules/datapipeline-lib/node_modules/node-rdkafka directory. At some point, make will produce a file named _tmp_build_fdd9e9cba78f819cac297390a51ef95a_SalesforceCloudServices_rbc_rebate_management_ace3bb7b6184ddfbad9427b727eb332cd26f01f3_node_modules_datapipeline_lib_node_modules_node_rdkafka_deps_librdkafka_gyp_librdkafka_target_build_dependencies.intermediate, which exceeds the maximum length of the filename on our OS (255 characters).

Would you have any ideas on where this problem could be fixed?

webmakersteve commented 6 years ago

Unfortunately I think this is just a problem with node-gyp. Since the paths are all so long, it is stripping the slashes and making unique files to act as locks of some sort. I'm not sure of a way to turn that functionality off. Have you used node-rdkafka in this project prior to the 2.3 update? Did it work then? I made some big changes to the way the library builds librdkafka so that may have caused some of the issues if that is the case. Otherwise, I'm not sure that there is a way to configure gyp to behave differently.

Is there any way you can avoid using the uuid in the build directories? Or make it shorter in heroku?

JesseDavid commented 6 years ago

I don't think there's a way we can avoid using the UUID in the build directories, as that's something built into the way Heroku manages it's internal build, but certainly something to bring up with their dev team.

We were able to get around this issue by renaming our repositories to shorter names, so it's not blocking at the moment.

We're using node-rdkafka @2.1.2, so it's nothing to do with the 2.3 update.

Thanks for the direction, we were thinking it's something in the node-gyp build process that may be causing this.

webmakersteve commented 6 years ago

Glad to hear you can get around the issue. Unfortunately part of the Makefile node-gyp generates creates files like that during the build process, and since we use submodules the paths can get long.

You may also be able to work around the problem by installing librdkafka from your distributions repositories and then linking against the system librdkafka.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.