Tundra doesn't respect 260 path limit on windows

[jimon]

If you use relative pathes in tundra.lua it's easy to end up in situation when absolute path to obj file is longer than 260 characters. In this case tundra doesn't complain until linker stage, when linking fails because linker cannot access that object file.

I can't find obvious workaround for this ... it's probably override function that generate path to obj files.

[leidegre]

Surely there must be something you can do to fix this issue with the way things are setup. Even if Windows happens to support longer paths. 260 characters is a lot of path information.

The way you work around this limitation in Windows is that you specify paths prefixed with \\?\.

So C:\foo.txt becomes \\?\C:\foo.txt. You can read about that here https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath

However, in my experience this is not really straightforward to do because it introduces additional complexity to paths in general. We don't expect rooted paths to look like \\?\C: and surely we do not really program for it. I've run into issues with tools simply not supported them, so even if tundra can would the compiler get it right? (Not sure about it, haven't tested it)

260 characters is a lot of path information, you sure you can't fix that with a symlink or renaming/reorganization of project files?

[jimon]

What I'm trying to do now is to change this line in path.lua :

    object_fn = "$(OBJECTDIR)/$(UNIT_PREFIX)/" .. relative_name .. "__" .. src_suffix .. suffix

to be something like this :

    object_fn = "$(OBJECTDIR)/$(UNIT_PREFIX)/" .. md5(relative_name) .. "__" .. src_suffix .. suffix

Maybe it will work just fine in my case, but I have hard time finding md5/sha/crc32 lua lib which works with lua setup in tundra :(

[jimon]

I failed to add any hash lib (all of them require bit/bit32 module, which is not available under tundra lua). So I'm doing this :

local relative_name = drop_suffix(object_fn:gsub("%.%.", "dotdot"))
path, relative_name2 = relative_name:match("(.-)([^\\/]-%.?([^%.\\/]*))$")
if relative_name2:len() > 0 then
  relative_name = "__" .. relative_name2
end

And getting this everytime :

LINK : fatal error LNK1181: cannot open input file '.obj'

Any ideas why is that ?

[leidegre]

Run the tundra2 binary with the verbose flag. This will render the command line as it is passed to the linker. It should reveal more information. There's also a Lua Debugger interface in there. There's a command to launch that (if you need to debug your Lua code).

Tundra uses and older Lua 5.1 and it does not support bit/bit32 module, these were added in Lua 5.2 (I think).

How come that you're relative paths are so long? What's causing this?

Note: It's kinda interesting what you're doing since it would turn the output folder to an content addressable store of some kind, except the content is the relative path name. It's gonna be annoying to find the outputs but other than that it could work.

There's a tundra.native.digest_guid function you could try. It's exposed by the tundra Lua environment. There's also a djb2_hash function. You'll find these in the src/LuaInterface.cpp file.

[deplinenoise]

Using a hash function seems like a reasonable workaround. (And as was pointed out, there's a MD5 function exposed you could use from Lua.)

The only drawback of hashing the paths is that you'll end up with a bunch of really opaque and debugging-unfriendly names. Maybe a scheme that retains some keyword (like the last directory name) followed by the hash could help a bit.

[jimon]

So I've finally got time to fix it, my solution is follows :

diff --git a/path.lua b/path.lua
index f5d7068..7381a35 100644
--- a/path.lua
+++ b/path.lua
@@ -1,6 +1,7 @@
module(..., package.seeall)

local npath = require "tundra.native.path"
+local native = require "tundra.native"

split             = npath.split
normalize         = npath.normalize
@@ -20,6 +21,14 @@ function remove_prefix(prefix, fn)
end
end

+function strip_unknown_symbols(str)
+  local s = ""
+  for i in str:gmatch( "%C+" ) do
+    s = s .. i
+  end
+  return s
+end
+
function make_object_filename(env, src_fn, suffix)
local object_fn

@@ -43,8 +52,9 @@ function make_object_filename(env, src_fn, suffix)
-- the DAG)
do
    local relative_name = drop_suffix(object_fn:gsub("%.%.", "dotdot"))
+    relative_name = strip_unknown_symbols(native.digest_guid(relative_name))
    object_fn = "$(OBJECTDIR)/$(UNIT_PREFIX)/" .. relative_name .. "__" .. src_suffix .. suffix
end

return object_fn
-end
+end
\ No newline at end of file

I had hard time previously because I haven't noticed that digest_guid add "\0" to string, so all appended strings "failed", because of this I had this error for a long time :

LINK : fatal error LNK1181: cannot open input file '.obj'

This solution is required if you have folder structure similar to :

C:/folder_a/name/name/.../name/name/test_1.cpp
C:/folder_a/name/name/.../name/name/test_2.cpp
...
C:/folder_a/name/name/.../name/name/test_N.cpp
C:/folder_b/name/name/.../name/name/tundra.lua

Because tundra.lua root folder is different (folder_b vs folder_a), tundra will create obj files with relative path, and this path will be longer then 260 characters.

Not sure if this solution is needed for upstream, but could be useful :)

[deplinenoise]

I had hard time previously because I haven't noticed that digest_guid add "\0" to string, so all appended strings "failed", because of this I had this error for a long time

Wow, that sounds like a different bug :) If we fix that too then there's no need for the slow pattern matching in Lua to fix things up.

I'll give some thought into how to best integrate this. Thanks!

[deplinenoise]

With f883d6e you should no longer see null characters. Can you rework your fix into a pull request? Thanks.

[jimon]

Sure, done :)