The matching result of the Glob function is wrong

[liruncong]

git repository: https://github.com/liruncong/GlobTest.git bsp: beaglebone mingw: https://download-sh-cmcc.rt-thread.org:9151/www/aozima/env_released_1%20.2.0.7z gcc position : env\tools\gnu_gcc\arm_gcc\mingw\bin python: 3.8.5(anaconda) platform: win11

As can be seen from the log, when Glob matches "D:\work\gitee\rt-thread_ninja\bsp\beaglebone\*\SConscript", the result is inconsistent with glob.glob, and there is more SConscript in the current directory. should not match to "D:\work\gitee\rt-thread_ninja\bsp\beaglebone\SConscript" file.

main test code:(D:\work\gitee\rt-thread_ninja\bsp\beaglebone\SConscript)

m = str(Dir('#')) + r"\*\SConscript"
print("m:", m)
print("<<<")
for s in Glob(m):
    print(str(s))
print(">>>")

print("<<<")
for s in glob.glob(m, recursive=True):
    print(str(s))
print(">>>")

log:

D:\work\gitee\rt-thread_ninja\bsp\beaglebone>set path=D:\MYBIN\scons-local;D:\MYBIN\Anaconda3\;D:\MYBIN\env\tools\gnu_gcc\arm_gcc\mingw\bin;%path%

D:\work\gitee\rt-thread_ninja\bsp\beaglebone>set RTT_EXEC_PATH=D:\MYBIN\env\tools\gnu_gcc\arm_gcc\mingw\bin

D:\work\gitee\rt-thread_ninja\bsp\beaglebone>set RTT_ROOT=D:\work\gitee\rt-thread_ninja

D:\work\gitee\rt-thread_ninja\bsp\beaglebone>scons
scons: Reading SConscript files ...
m: D:\work\gitee\rt-thread_ninja\bsp\beaglebone\*\SConscript
<<<
D:\work\gitee\rt-thread_ninja\bsp\beaglebone\SConscript
D:\work\gitee\rt-thread_ninja\bsp\beaglebone\applications\SConscript
D:\work\gitee\rt-thread_ninja\bsp\beaglebone\drivers\SConscript
>>>
<<<
D:\work\gitee\rt-thread_ninja\bsp\beaglebone\applications\SConscript
D:\work\gitee\rt-thread_ninja\bsp\beaglebone\drivers\SConscript
>>>
target= Test.autogen.cpp
Newlib version:2.4.0
scons: done reading SConscript files.
scons: Building targets ...
scons: building associated VariantDir targets: build
LINK rtthread-beaglebone.elf
arm-none-eabi-objcopy -O binary rtthread-beaglebone.elf rtthread.bin
arm-none-eabi-size rtthread-beaglebone.elf
   text    data     bss     dec     hex filename
 136737    1912    2660  141309   227fd rtthread-beaglebone.elf
scons: done building targets.

[bdbaddog]

You seem to keep ignoring the request in the form for filing bugs that you bring these issues to our discord server or the users mailing list.

Anyway the underlying logic is this:

from fnmatch import fnmatch

dirlist = [
r"D:\work\gitee\rt-thread_ninja\bsp\beaglebone\SConscript",
r"D:\work\gitee\rt-thread_ninja\bsp\beaglebone\applications\SConscript",
r"D:\work\gitee\rt-thread_ninja\bsp\beaglebone\drivers\SConscript",
]

pattern = r'D:\work\gitee\rt-thread_ninja\bsp\beaglebone\*\SConscript'

for d in dirlist:
    matches = fnmatch(d, pattern)
    print("D[Matches:%s]:%s"%(matches,d))

Which yields:

% python3.10 test.py
D[Matches:False]:D:\work\gitee\rt-thread_ninja\bsp\beaglebone\SConscript
D[Matches:True]:D:\work\gitee\rt-thread_ninja\bsp\beaglebone\applications\SConscript
D[Matches:True]:D:\work\gitee\rt-thread_ninja\bsp\beaglebone\drivers\SConscript

Which matches the results you got. So this is indeed the expected functionality.

SCons cannot use python's glob.glob since that requires files to exist in the filesystem. SCons' Glob() can see both files in the filesystem and files which SCons knows how to build but aren't yet built.

Likely we should update the docs to indicate that the underlying logic is supplied by python's fnmatch.fnmatch ? (@mwichmann thoughts)

[liruncong]

I have successfully applied to join the mailing list, sorry I didn't notice this requirement before.

I see the result of your fnmatch execution above, the first one is False, which means no match. This means that only two files should be matched. However, after Glob is actually executed, three files are matched (3 files are printed in the log), which is the problem. And glob.glob matches two files (two files are printed in the log), which is consistent with the fnmatch result.

[bdbaddog]

Ahh.. You're correct. I'll take a further look at the code..

[mwichmann]

Likely we should update the docs to indicate that the underlying logic is supplied by python's fnmatch.fnmatch ? (@mwichmann thoughts)

It does say that, but perhaps it's only in the function docstring, where it says:

The underlying algorithm is adapted from the glob.glob() function in the Python library (but heavily modified), and uses fnmatch() under the covers.

I'm not sure exactly that statement would be appropriate in the manpage, sounds like an implementation detail, but maybe there's something else that should be said.

Glob has multiple differences from glob.glob, which I think has been said already (it's certainly in the manpage): if matches in-memory nodes that may not exist in the filesystem, it understands the backing Repository stuff, and it understands variant directories, and it does substitution, and it has an exclude pattern. I was looking at Glob because of the desire to add recursion (#3851), and it is rather convoluted, sadly...

[liruncong]

@bdbaddog Hi, how to reopen an issue? #4118 didn't solve the problem, but I don't know how to reopen the issue.

[bdbaddog]

@bdbaddog Hi, how to reopen an issue? #4118 didn't solve the problem, but I don't know how to reopen the issue.

Why comment on #4118 here?

[liruncong]

I don't know if you would notice if I commented on a closed issue.

[mwichmann]

Just swung back to this because I'm otherwise working on Glob, and I can't seem to reproduce this - making this temporary SConstruct file in an rt-thread checkout, and the matching seems to be as expected - sconscripts only in the subdirs. Odd.

[liruncong]

@mwichmann I can still reproduce the problem.

[mwichmann]

Even tried it on a Windows machine in case it's somehow different there...

scons: Reading SConscript files ...
pattern: .\*\SConscript
<<< Glob
applications\SConscript
drivers\SConscript
>>>
<<< glob.glob
.\applications\SConscript
.\drivers\SConscript
>>>

[liruncong]

@mwichmann Could it be related to "Dir('#')"?

[mwichmann]

Still not able to reproduce. Meanwhile - what's your objective here? If you just want to generate file names, you can use GetLaunchDir instead of doing a dance with making a node out of # and then converting it to a string. Glob can also be asked to produce strings if that's what you're after (Glob(pattern, strings=True)).