python setup.py install works without git submodule being initialized

[mithro]

I forgot to init the submodules when cloning misoc. Running python setup.py install worked fine but I later got an error like the following when trying to build;

 CC       crc16.o                                                                                                                                                                                        [7700/9787]
 CC       crc32.o
 CC       console.o
 CC       system.o
 CC       id.o
 CC       uart.o
 CC       time.o
 CC       qsort.o
 CC       strtod.o
 CC       spiflash.o
 CC       vsnprintf.o
 AR       libbase.a
 CC       vsnprintf-nofloat.o
 AR       libbase-nofloat.a
make: Leaving directory `/usr/local/google/home/tansell/foss/timvideos/hdmi2usb/i2cslave/misoc_i2csoc_pipistrello_i2c/software/libbase'
make: Entering directory `/usr/local/google/home/tansell/foss/timvideos/hdmi2usb/i2cslave/misoc_i2csoc_pipistrello_i2c/software/libcompiler_rt'
make: *** No rule to make target `divsi3.o', needed by `libcompiler_rt.a'.  Stop.
make: Leaving directory `/usr/local/google/home/tansell/foss/timvideos/hdmi2usb/i2cslave/misoc_i2csoc_pipistrello_i2c/software/libcompiler_rt'
Traceback (most recent call last):
  File "/usr/local/google/home/tansell/foss/timvideos/hdmi2usb/i2cslave/build/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/google/home/tansell/foss/timvideos/hdmi2usb/i2cslave/build/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/google/home/tansell/foss/timvideos/hdmi2usb/i2cslave/i2cslave/targets/pipistrello_i2c.py", line 403, in <module>
    main()
  File "/usr/local/google/home/tansell/foss/timvideos/hdmi2usb/i2cslave/i2cslave/targets/pipistrello_i2c.py", line 399, in main
    builder.build()
  File "/usr/local/google/home/tansell/foss/timvideos/hdmi2usb/i2cslave/build/lib/python3.4/site-packages/misoc-0.1-py3.4.egg/misoc/integration/builder.py", line 133, in build
    self._generate_software()
  File "/usr/local/google/home/tansell/foss/timvideos/hdmi2usb/i2cslave/build/lib/python3.4/site-packages/misoc-0.1-py3.4.egg/misoc/integration/builder.py", line 110, in _generate_software
    subprocess.check_call(["make", "-C", dst_dir])
  File "/usr/local/google/home/tansell/foss/timvideos/hdmi2usb/i2cslave/build/lib/python3.4/subprocess.py", line 561, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-C', '/usr/local/google/home/tansell/foss/timvideos/hdmi2usb/i2cslave/misoc_i2csoc_pipistrello_i2c/software/libcompiler_rt']' returned non-zero exit status 2
tansell@tansell-z620-l2:~/foss/timvideos/hdmi2usb/i2cslave$ ls

Running git submodule update --init in the misoc directory and then python setup.py install again fixed the problem.

python setup.py install should probably complain if it can't find the needed files from submodules.

[sbourdeauducq]

@mithro Do you want to fix this, and thoroughly test that you have not broken anything? setuptools is brimming with ridiculous design flaws, and doing seemingly simple things like this often means opening a whole can of worms.

[mithro]

This is the code I came up with for litex's setup.py to check this.

def check_submodules(basedir):
    try:
        modules = open(os.path.join(basedir, ".gitmodules")).read()
    except FileNotFoundError as e:
        return []

    submodule_errors = []
    for line in modules.splitlines():
        match = re.search("path = (.+)", line)
        if not match:
            continue

        modulepath = match.groups()[0]

        fullpath = os.path.normpath(os.path.join(basedir, modulepath))
        assert os.path.exists(fullpath)
        assert os.path.isdir(fullpath)
        if not os.path.exists(os.path.join(fullpath, ".git")):
            submodule_errors.append(fullpath)
            continue
        submodule_errors += check_submodules(fullpath)
    return submodule_errors

if sys.version_info[:3] < (3, 3):
    raise SystemExit("You need Python 3.3+")

submodule_errors = check_submodules(os.path.dirname(__file__))
if submodule_errors:
    raise SystemExit("""\
The following git submodules are not initialized:{}

Please run:
git submodule update --recursive --init
git submodule foreach git submodule update --recursive --init
""".format("\n * ".join([""]+submodule_errors)))

Do you think something similar would be suitable for misoc?

[sbourdeauducq]

Parsing .gitmodules is fragile (when done like that) and complicated. How about checking the output of git submodule status? It will even tell you directly if the submodule is initialized.

[whitequark]

I would say it is not especially fragile if fixed to ignore whitespace, and take end-of-line/beginning-of-line into account. I.e. r"^\w*path\w*=\w*(.+)", since that captures the actual grammar used by git.

[mithro]

I think launching a subprocess (and parsing its output) is likely to be more fragile than parsing the .gitmodule files but happy to do which ever version you prefer.