Open firewave opened 5 years ago
The does not occur when you build it in a Docker container which is running ubuntu 16.04, but the host system is not ubuntu 16.04. If both the container and the host run it the issue occurs.
On a different system the last log messages differed
CAUTION: dependent file libs/sort/doc/graph/windows_string_sort_files/editdata does not exist.
Found while scanning file libs/sort/doc/graph/windows_string_sort_files/chart001.htm
CAUTION: dependent file libs/sort/doc/graph/windows_string_sort_files/stylesheet does not exist.
Found while scanning file libs/sort/doc/graph/windows_string_sort_files/chart001.htm
CAUTION: dependent file libs/sort/doc/graph/windows_string_sort_files/image001 does not exist.
Found while scanning file libs/sort/doc/graph/windows_string_sort_files/chart001.htm
Segmentation fault (core dumped)
I'm seeing this too in a 16.04 docker hosted in 18.04
I'm unable to reproduce on either MSVC or Ubunutu-19.10, so I will need more information.
Infinite recursion in add_path
is prevented by (add_path.cpp:96):
if(m_copy_paths.find(p) != m_copy_paths.end())
return;
So for some reason the the check for "have we seen this file before" is failing, even though I assume we really do already have it in the list. Are you able to debug and see what the argument to add_path
is that causes the issue? My guess is that there's a file with some sort of screwball name somewhere that is confusing Boost.Filesystem.
Running in docker might be essential to reproducing the issue.
I'm running a 16.04 docker inside a 18.04 host, but it might be possible to reproduce without those exact versions.
Here are the commands that lead to the segfault:
tar -xjf boost_1_71_0.tar.bz2
cd boost_1_71_0
./bootstrap.sh
./b2 tools/bcp/
mkdir nsboost
./dist/bin/bcp libs ./nsboost
The thing is you're talking to a non-linux non-docker user here.
The output will be voluminous, but what happens if you add:
std::cout << p << std::endl;
To the start of bcp_implementation::add_file
, does this show what file is triggering the infinite recursion?
BTW the only other situation I can think of that would lead to this, is if there is a link leading to a circular filesystem path - that should obviously not be the case within the boost tree.
Yes, I added content similar to that cout in a few of the methods there. It doesn't seem to be infinite recursion to me, but I was not able to narrow down the test case. It seems to be non-deterministic to some extent.
What was the final output before the overflow?
This is not possible to debug interactively through the medium of github comments.
Please try to get access to a linux vm or machine and reproduce it locally with docker.
What was the final output before the overflow?
The final output is non-deterministic. It is a different output depending on how much I try to trim the files that bcp processes.
I think what's happening is that the limits of Ubuntu 16.04 are being hit somehow. I can't reproduce this in an 18.04 docker, only a 16.04 one.
I added a cout line to add_path and I got 65049
lines before the segfault. That's suspiciously close to 2^16
, and I think bcp
adds each path entry to a container. My guess is that the compiler or the std library on 16.04 has a limit to how many items can go into the container, and the 1.71 release hits that limit.
This isn't an infinite recursion problem.
This patch excludes enough files to make it possible for me to run bcp on 16.04:
--- a/add_path.cpp
+++ b/add_path.cpp
@@ -40,6 +40,9 @@ void bcp_implementation::add_path(const fs::path& p)
void bcp_implementation::add_directory(const fs::path& p)
{
+ if (p.leaf() == "html")
+ return;
+
//
// Don't add files created by build system:
//
This also proves that the issue is somehow about processing too many files. With this patch only 42650 files are processed.
This is probably caused by too many active fileview
s and running out of memory as a consequence. (Scanning an html
file and add_file_dependencies
both call add_path
while a fileview
is still live.)
Can you please try again with current develop? I've changed it to a fully non-recursive algorithm, so stack usage is now low. The catch is that a mighty big list of files gets constructed, so dynamic memory usage goes up to ~ 80Mb if you end up pulling in everything.
I applied the changes to Boost 1.70 and my build was successful.
Using https://github.com/conan-community/conan-boost with
boost:namespace = boost_ns
andboost:namespace_alias = True
will fail with Boost 1.70 on ubuntu 16.04 with a core dump.The stack trace looks like this and goes on with the same two function calls
The bcp command-line
It worked fine with Boost 1.69. It also works with 1.70 on CentOS 7, ubuntu 18.04, ubuntu 19.04 and Kali 2019.2.
I am using the default compiler on a x86_64 machine.