Can't build project that contains two files with identical name

dawidchyrzynski commented 2 years ago

Hi,

My project uses ESP8266 core and Arduino Crypto library (https://github.com/OperatorFoundation/Crypto). The problem is that both repositories contain "Crypto.h" and "Crypto.cpp" files. Due to flat structure of the build directory Crypto.cpp from ESP8266 overrides Crypto.cpp from Arduino Crypto and it leads to linking problems (missing symbols). Is there any way to solve this issue (assuming that we're not going to modify third party sources)?

everslick commented 2 years ago

This is a looming issue for me as well for a very long time. I had to work around name clashes in third party libs, most often by forking them and renaming individual files. Building on Mac makes things even worse due to case insensitive file names (by default). I don't know how feasable it would be to keep the directory structure for the object files during the build though.

plerup commented 2 years ago

Ok, point taken. I will look into creating sub directories in the build directory

plerup commented 2 years ago

I have tried several solutions to this problem but selected the easiest one. Please try out the latest commit

dawidchyrzynski commented 2 years ago

Hi @plerup,

Unfortunately it didn't solve the problem. I can see two files being compiled (one with original name, the second one with "2_" prefix) but linker still complains about missing symbols. When I'm adding prefix manually to the original cpp file the build succeeds without any issue.

plerup commented 2 years ago

Is it possible for you to share these files with me?

hamishcunningham commented 2 years ago

hi Peter,

I've just seen the same problem with commit f30b6fe: the second file duplicating a name gets compiled (as 2_...) but doesn't get linked. Renaming the file works.

I first thought that the src_list.mk additions in find_src.pl need to also add the renamed source file to a list of objects to link, but the problem seems to be influenced by execution order. Turning off both make parallelism and ccache seems to help.

My build is here: https://gitlab.com/hamishcunningham/unphone/-/tree/master/examples/2022 The issue manifests in relation to WebServer.cpp in the ESPAsyncWebServer library.

Tnx! Hamish

hamishcunningham commented 2 years ago

Also, if any source files are symbolic links, they get renamed (even though they're unique).

plerup commented 2 years ago

@hamishcunningham Please try setting NO_USER_OBJ_LIB=1

hamishcunningham commented 2 years ago

@hamishcunningham Please try setting NO_USER_OBJ_LIB=1

This appears to work :)

Observations:

putting NO_USER_OBJ_LIB = 1 in my Makefile (before including makeEspArduino.mk) doesn't seem to work: I have to put it on the command line
the problem is intermittent, so probably an order of execution issue; this makes me suspect that sometimes the source code of the duplicate is copied to both the original name and the 2_... name, meaning that the duplicate is never compiled? perhaps if the ordering was restricted somehow in the perl generator script it would help?
using BUILD_THREADS=1 also seems to work, supporting the order of execution theory (though again the intermittent nature means it is hard to be absolutely sure)

Tnx!

plerup commented 2 years ago

@hamishcunningham Tried building your example at https://gitlab.com/hamishcunningham/unphone/-/tree/master/examples/2022 and that works without any problem but then again there are no duplicates in this case

I had to create a dummy private.h with the definitions for SSID and PSK, should it contain something more?

hamishcunningham commented 2 years ago

@plerup thanks! That example should generate two copies of WebServer.cpp, one from the AsyncWebServer library and one from the Arduino WebServer library...? Then my error is that the linker sometimes can't find the AsyncWebServer symbols.

(SSID and PSK defs should be all you need in private.h)

hamishcunningham commented 2 years ago

PS I guess that if you don't have this repo https://gitlab.com/hamishcunningham/the-internet-of-things also checked out in $HOME and haven't run support/magic.sh setup then the latter copy of WebServer won't be present

plerup commented 2 years ago

@hamishcunningham

Did this but still no duplicates or problems. This is what I did in a clean virtual Ubuntu 20.04

git clone https://gitlab.com/hamishcunningham/the-internet-of-things.git git clone https://gitlab.com/hamishcunningham/unphone.git cd the-internet-of-things/ support/magic.sh setup cd cd unphone/examples/2022 ./../../the-internet-of-things/support/magic.sh mkesp

Please advice

hamishcunningham commented 2 years ago

Sorry, I updated the libraries and also renamed the clashing file. If you move it back:

mv ~/unphone/lib/ESPAsyncWebServer/src/XXXWebServer.cpp ~/unphone/lib/ESPAsyncWebServer/src/WebServer.cpp

Then you will get the 2_WebServer.cpp file appearing in /tmp/mkESP/sketch_featheresp32.

Subsequent builds fail intermittently with duplicate or missing symbol linker errors, e.g.

~/unphone/examples/2022 $ make -f ~/the-internet-of-things/support/tooling/makeEspArduino/makeEspArduino.mk ESP_ROOT=~/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32     BOARD=featheresp32 CHIP=esp32 LIBS=~/unphone/lib  SKETCH=sketch/sketch.ino all
Linking /tmp/mkESP/sketch_featheresp32/sketch.bin
  Versions: cdd4aa0-dirty, 2.0.2
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o):(.literal._Z13initWebServerv+0x38): undefined reference to `AsyncWebServer::on(char const*, unsigned char, std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o):(.literal._Z13initWebServerv+0x3c): undefined reference to `AsyncWebServer::onNotFound(std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o):(.literal._Z13initWebServerv+0x40): undefined reference to `AsyncWebServer::begin()'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o):(.literal.startup._GLOBAL__sub_I__Z7getHtmlR6StringPPKciP13replacement_ti+0x4): undefined reference to `AsyncWebServer::AsyncWebServer(unsigned short)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o):(.literal.exit._GLOBAL__sub_D__Z7getHtmlR6StringPPKciP13replacement_ti+0x0): undefined reference to `AsyncWebServer::~AsyncWebServer()'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o): in function `initWebServer()':
/home/hamish/unphone/examples/2022/sketch/httpd.cpp:176: undefined reference to `AsyncWebServer::on(char const*, unsigned char, std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/hamish/unphone/examples/2022/sketch/httpd.cpp:188: undefined reference to `AsyncWebServer::on(char const*, unsigned char, std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/hamish/unphone/examples/2022/sketch/httpd.cpp:199: undefined reference to `AsyncWebServer::on(char const*, unsigned char, std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/hamish/unphone/examples/2022/sketch/httpd.cpp:201: undefined reference to `AsyncWebServer::onNotFound(std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/hamish/unphone/examples/2022/sketch/httpd.cpp:203: undefined reference to `AsyncWebServer::begin()'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o): in function `_GLOBAL__sub_I__Z7getHtmlR6StringPPKciP13replacement_ti':
/home/hamish/unphone/examples/2022/sketch/httpd.cpp:113: undefined reference to `AsyncWebServer::AsyncWebServer(unsigned short)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o): in function `_GLOBAL__sub_D__Z7getHtmlR6StringPPKciP13replacement_ti':
/home/hamish/unphone/examples/2022/sketch/httpd.cpp:113: undefined reference to `AsyncWebServer::~AsyncWebServer()'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(WebRequest.cpp.o):(.literal._ZN21AsyncWebServerRequest13_onDisconnectEv+0x0): undefined reference to `AsyncWebServer::_handleDisconnect(AsyncWebServerRequest*)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(WebRequest.cpp.o): in function `AsyncWebServerRequest::_onDisconnect()':
/home/hamish/unphone/lib/ESPAsyncWebServer/src/WebRequest.cpp:229: undefined reference to `AsyncWebServer::_handleDisconnect(AsyncWebServerRequest*)'
collect2: error: ld returned 1 exit status
make: *** [/home/hamish/the-internet-of-things/support/tooling/makeEspArduino/makeEspArduino.mk:313: /tmp/mkESP/sketch_featheresp32/sketch.bin] Error 1

It isn't consistent :( but I've also seen it in other similar builds.

What appears to have happened is that the file from the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/libraries/WebServer/src/WebServer.cpp has been copied into 2_WebServer.cpp, whereas it should have been copied from unphone/lib/ESPAsyncWebServer/src/WebServer.cpp.

Perhaps make is interleaving two tasks during parallel execution, or the script that sets up the copy is backgrounded, or similar?

hamishcunningham commented 2 years ago

Or perhaps which copy of WebServer.cpp is reached first during the build gets copied into 2_WebServer.cpp, and sometimes this is from one tree and sometimes the other? (A fix might be for the perl script to order the list of candidates by full path before processing?)

plerup commented 2 years ago

Thanks, this example made me realize that the current solution for handling clashing file names is only working in some cases. I will have to implement a new one. Stay tuned...

plerup commented 2 years ago

@hamishcunningham Please test latest commit

hamishcunningham commented 2 years ago

@plerup have only done a brief test but seems to work so far :) (I can see both 1... and 2... versions of the duplicate being compiled.) Thanks! Will try more tomorrow.

hamishcunningham commented 2 years ago

Pretty sure this works now :) Can be closed from my point-of-view, thanks!

plerup / makeEspArduino

Can't build project that contains two files with identical name #174