Doesn't build with setup.py build

[fccoelho]

I have a small Pyd project with a single module which builds and runs normally with dub, but is raising the following error when I try to build it with python setup.py build:

source/models.d(9): Error: module `variable` is in file 'mir/random/variable.d' which cannot be read
import path[0] = /usr/local/lib/python3.7/dist-packages/pyd/infrastructure
import path[1] = /snap/dmd/74/bin/../import/druntime
import path[2] = /snap/dmd/74/bin/../import/phobos
travis_fold:end:pyd_compile-e84aa9d4-ded9-11e9-936e-342eb6905833
error: command 'dmd' failed with exit status 1

It seems that it can't find the variable package from within mir

Is there another way to build it with dub into a library which can be imported from Python?

[fccoelho]

Here is What my setup.py looks like:

from pyd.support import setup, Extension

projName = "Epidemiad"

setup(
    name=projName,
    version='0.1',
    ext_modules=[
        Extension(projName, ['source/models.d'],
            extra_compile_args=['-w'],
            build_deimos=True,
            d_lump=True
        )
    ],
)

and here is my dub.conf:

{
    "authors": [
        "Flávio Codeco Coelho"
    ],
    "copyright": "Copyright © 2019, Flávio Codeco Coelho",
    "dependencies": {
        "mir": "~>3.2.0",
        "mir-algorithm": "~>3.5.5",
        "mir-random": "~>2.2.6",
        "pyd": "~>0.12.0"
    },
    "description": "Stochastic SIR model in D",
    "license": "MIT",
    "name": "epidemiad",
    "sourcePaths": [
        "source/"
    ],
    "subConfigurations": {
            "pyd": "python37"
    }
}

[ariovistus]

looks like Extension accepts arguments libraries, include_dirs, and library_dirs as lists of strings. these would correspond to -l -I -L on the dmd command line, so if you can figure out what dub is feeding your compiler, it might be possible to get it building with setup.py Can't say I've ever tried this though

[fccoelho]

I'll give it a shot, but it would be nice if Extension could get that info straight from dub.conf, when it's available. Do you think that would be feasible? this would make it much easier for newcomers to PyD like me.

[fccoelho]

I have added my ~/.dub/packages directory to the include dirs, and it still can't find the variable.d module.

any other suggestions?

[fccoelho]

I got it to build after adding various Mir related directories to include_dirs here is how my setup.py ended up like

from pyd.support import setup, Extension

projName = "Epidemiad"

setup(
    name=projName,
    version='0.1',
    ext_modules=[
        Extension(projName, ['source/models.d'],
                  include_dirs=['~/.dub/packages/mir-random-2.2.6/mir-random/source',
                                '~/.dub/packages/mir-core-1.0.2/mir-core/source/',
                                '~/.dub/packages/mir-3.2.0/mir/source/',
                                '~/.dub/packages/mir-linux-kernel-1.0.1/mir-linux-kernel/source/',
                                '~/.dub/packages/mir-algorithm-3.5.5/mir-algorithm/source/'
                                ],
                  # extra_compile_args=['-w'],
                  build_deimos=True,
                  d_lump=True
                  )
    ],
)

but now I am getting the the following error when I try to import it:

In [1]: import Epidemiad                                                                                                                                               
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-a8bc7d58532c> in <module>
----> 1 import Epidemiad

ImportError: /home/fccoelho/Documentos/Aulas/FGV_Programming_Languages/D/Exercicios/Epidemia/epidemiad/build/lib.linux-x86_64-3.7/Epidemiad.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _D3mir6random12__ModuleInfoZ

[fccoelho]

If this simple example gets sorted it can become a good example of using PyD in a project making use of Mir. ;-)

[ariovistus]

seems like there ought to be library references whenever there are include directories. the latter tells the compile step that the included symbols exist, but the link step needs the libraries. I haven't used dub in a while, I'm assuming it isn't just globbing the sources of all the dependencies into the compile of your application. dub should have a verbose flag ya? you should be able to find the library parameters from its output

[fccoelho]

dub build --force -v spits out a ton of lines with are hard to read.

dub describe provides a JSON with lot of information, which is much easier to parse. It is too much to paste here, but I am pasting some of it bellow:

{
...
"targets": [
                {
                        "rootPackage": "epidemiad",
                        "packages": [
                                "epidemiad",
                                "mir"
                        ],
                        "rootConfiguration": "application",
                        "buildSettings": {
                                "targetType": 2,
                                "targetPath": "/home/fccoelho/Documentos/Aulas/FGV_Programming_Languages/D/Exercicios/Epidemia/epidemiad",
                                "targetName": "epidemiad",
                                "workingDirectory": "",
                                "mainSourceFile": "/home/fccoelho/Documentos/Aulas/FGV_Programming_Languages/D/Exercicios/Epidemia/epidemiad/source/app.d",
                                "dflags": [],
                                "lflags": [],
                                "libs": [
                                        "python3.7m"
                                ],
                                "linkerFiles": [
                                        "/home/fccoelho/.dub/packages/mir-random-2.2.6/mir-random/libmir-random.a",
                                        "/home/fccoelho/.dub/packages/mir-algorithm-3.5.6/mir-algorithm/libmir-algorithm.a",
                                        "/home/fccoelho/.dub/packages/mir-core-1.0.2/mir-core/libmir-core.a",
                                        "/home/fccoelho/.dub/packages/mir-linux-kernel-1.0.1/mir-linux-kernel/libmir-linux-kernel.a",
                                        "/home/fccoelho/.dub/packages/pyd-0.12.0/pyd/libpyd.a"
                                ],
                                "sourceFiles": [
                                        "/home/fccoelho/Documentos/Aulas/FGV_Programming_Languages/D/Exercicios/Epidemia/epidemiad/source/app.d",
                                        "/home/fccoelho/Documentos/Aulas/FGV_Programming_Languages/D/Exercicios/Epidemia/epidemiad/source/models.d",
                                        "/home/fccoelho/.dub/packages/mir-3.2.0/mir/source/mir/glas/l1.d",
                                        "/home/fccoelho/.dub/packages/mir-3.2.0/mir/source/mir/glas/l2.d",
                                        "/home/fccoelho/.dub/packages/mir-3.2.0/mir/source/mir/glas/package.d",
                                        "/home/fccoelho/.dub/packages/mir-3.2.0/mir/source/mir/model/lda/hoffman.d",
                                        "/home/fccoelho/.dub/packages/mir-3.2.0/mir/source/mir/sparse/blas/axpy.d",
                                        "/home/fccoelho/.dub/packages/mir-3.2.0/mir/source/mir/sparse/blas/dot.d",
                                        "/home/fccoelho/.dub/packages/mir-3.2.0/mir/source/mir/sparse/blas/gemm.d",
                                        "/home/fccoelho/.dub/packages/mir-3.2.0/mir/source/mir/sparse/blas/gemv.d",
                                        "/home/fccoelho/.dub/packages/mir-3.2.0/mir/source/mir/sparse/blas/package.d",
                                        "/home/fccoelho/.dub/packages/mir-3.2.0/mir/source/mir/sparse/package.d"
                                ],
...
}

I tried using the linker files above as libraries but it didn't work. I can't seem to figure this out. I anyone wants to give it a try, the project is here

[ariovistus]

seems you want to put those linkerFiles in the parameter extra_link_args like so:

setup(
    name=projName,
    version='0.1',
    ext_modules=[
        Extension(projName, ['source/models.d'],
                  include_dirs=['~/.dub/packages/mir-random-2.2.6/mir-random/source',
                                '~/.dub/packages/mir-core-1.0.2/mir-core/source/',
                                '~/.dub/packages/mir-3.2.0/mir/source/',
                                '~/.dub/packages/mir-linux-kernel-1.0.1/mir-linux-kernel/source/',
                                '~/.dub/packages/mir-algorithm-3.5.6/mir-algorithm/source/'
                                ],
                  #extra_compile_args=['', ''],
                  extra_link_args=[
                    "/home/ellery/.dub/packages/mir-random-2.2.6/mir-random/libmir-random.a",
                    "/home/ellery/.dub/packages/mir-algorithm-3.5.6/mir-algorithm/libmir-algorithm.a",
                    "/home/ellery/.dub/packages/mir-core-1.0.2/mir-core/libmir-core.a",
                    "/home/ellery/.dub/packages/mir-linux-kernel-1.0.1/mir-linux-kernel/libmir-linux-kernel.a"
                  ],
                  build_deimos=True,
                  d_lump=True
                  )
    ],
)

[fccoelho]

Thanks a lot @ariovistus ! it worked! to automate it, I came up with the following updated setup.py:

from pyd.support import setup, Extension
import os, json

f = os.popen('dub describe')
build_pars = json.loads(f.read())['targets'][0]['buildSettings']

projName = "epidemiad"

setup(
    name=projName,
    version='0.1',
    ext_modules=[
        Extension(projName, ['source/models.d'],
                  include_dirs=build_pars['importPaths'],
                  extra_link_args=build_pars['linkerFiles'],
                  # extra_compile_args=['-w'],
                  build_deimos=True,
                  d_lump=True
                  )
    ],
)

Now I can import the library which is built with python setup.py build. By using Dub's build information automatically like this, I believe that now other users can build D extensions for Python using any Dub package without having to fiddle with building parameters.