Closed fcannini closed 3 years ago
About the first one: can you try again by replacing this line:
https://github.com/devitocodes/devito/blob/master/devito/compiler.py#L557
with
obj = super().__new__(cls)
?
if it works, could you open a PR applying this simple patch? Guidelines available here
About the other set of issues. The key thing to notice is:
E OSError: /tmp/devito-jitcache-uid927800007/d7c6f77c0044b47f33dd888e61168861bcb53547.so: undefined symbol: omp_get_thread_num
It appears to be unable to compile OpenMP.
I think the actual issue is that the system you're running on has some preset CC, CFLAGS, CXX or LDFLAGS environment variables, so Devito won't use its own preset flags to compile the generated code, which may or may not include OpenMP (those tests do). You could either unset those flags in your environment, or tell Devito to use a very specific compiler. Assuming you're on bash, you could
export DEVITO_ARCH=gcc-9.2.0
which hopefully should make the trick
And if you got that far, I'd assume you didn't encounter the ARM related issues reported here. Right?
@FabioLuporini The first one worked just fine, thank you. Now all test_operator.py
tests are passing.
The second issue remains, however. When runing with DEVITO_ARCH=gcc
, the error message now is the following for all the same tests (with minor differences like the amount of variables).
self = <test_dse.TestAliases object at 0x400041992898>
def test_full_shape(self):
"""
Check the shape of the Array used to store an aliasing expression.
The shape is impacted by loop blocking, which reduces the required
write-to space.
"""
grid = Grid(shape=(3, 3, 3))
x, y, z = grid.dimensions # noqa
t = grid.stepping_dim
f = Function(name='f', grid=grid)
f.data_with_halo[:] = 1.
u = TimeFunction(name='u', grid=grid, space_order=3)
u.data_with_halo[:] = 0.5
# Leads to 3D aliases
eqn = Eq(u.forward, ((u[t, x, y, z] + u[t, x+1, y+1, z+1])*3*f +
(u[t, x+2, y+2, z+2] + u[t, x+3, y+3, z+3])*3*f + 1))
op0 = Operator(eqn, opt=('noop', {'openmp': True}))
op1 = Operator(eqn, opt=('advanced', {'openmp': True, 'cire-mincost-sops': 1}))
x0_blk_size = op1.parameters[2]
y0_blk_size = op1.parameters[3]
z_size = op1.parameters[4]
# Check Array shape
arrays = [i for i in FindSymbols().visit(op1._func_table['bf0'].root)
if i.is_Array and i._mem_local]
assert len(arrays) == 1
a = arrays[0]
assert len(a.dimensions) == 3
assert a.halo == ((1, 1), (1, 1), (1, 1))
assert Add(*a.symbolic_shape[0].args) == x0_blk_size + 2
assert Add(*a.symbolic_shape[1].args) == y0_blk_size + 2
assert Add(*a.symbolic_shape[2].args) == z_size + 2
# Check numerical output
op0(time_M=1)
exp = np.copy(u.data[:])
u.data_with_halo[:] = 0.5
> op1(time_M=1)
test_dse.py:431:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/spack/opt/spack/linux-centos7-thunderx2/gcc-9.2.0/py-devito-4.2.3-7k3p5kb7krot6dzife2q2irstrs2apji/lib/python3.6/site-packages/devito/operator/operator.py:603: in __call__
return self.apply(**kwargs)
/opt/spack/opt/spack/linux-centos7-thunderx2/gcc-9.2.0/py-devito-4.2.3-7k3p5kb7krot6dzife2q2irstrs2apji/lib/python3.6/site-packages/devito/operator/operator.py:674: in apply
cfunction = self.cfunction
/opt/spack/opt/spack/linux-centos7-thunderx2/gcc-9.2.0/py-devito-4.2.3-7k3p5kb7krot6dzife2q2irstrs2apji/lib/python3.6/site-packages/devito/operator/operator.py:589: in cfunction
self._jit_compile()
/opt/spack/opt/spack/linux-centos7-thunderx2/gcc-9.2.0/py-devito-4.2.3-7k3p5kb7krot6dzife2q2irstrs2apji/lib/python3.6/site-packages/devito/operator/operator.py:575: in _jit_compile
str(self.ccode))
/opt/spack/opt/spack/linux-centos7-thunderx2/gcc-9.2.0/py-devito-4.2.3-7k3p5kb7krot6dzife2q2irstrs2apji/lib/python3.6/site-packages/devito/compiler.py:318: in jit_compile
sleep_delay=sleep_delay)
/opt/spack/opt/spack/linux-centos7-thunderx2/gcc-9.2.0/py-codepy-2019.1-xm7xfwu5rxzzzoxidhll3tzlnkg2zj5w/lib/python3.6/site-packages/codepy/jit.py:433: in compile_from_string
toolchain.build_extension(ext_file, source_paths, debug=debug)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = JITCompiler[GNUCompiler]
ext_file = '/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.so'
source_files = ['/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c']
debug = False
def build_extension(self, ext_file, source_files, debug=False):
cc_cmdline = (
self._cmdline(source_files, False)
+ ["-o", ext_file]
)
from pytools.prefork import call
if debug:
print(" ".join(cc_cmdline))
result = call(cc_cmdline)
if result != 0:
import sys
print("FAILED compiler invocation:" + " ".join(cc_cmdline),
file=sys.stderr)
> raise CompileError("module compilation failed")
E codepy.CompileError: module compilation failed
/opt/spack/opt/spack/linux-centos7-thunderx2/gcc-9.2.0/py-codepy-2019.1-xm7xfwu5rxzzzoxidhll3tzlnkg2zj5w/lib/python3.6/site-packages/codepy/toolchain.py:210: CompileError
----------------------------- Captured stderr call -----------------------------
Operator `Kernel` run in 0.01 s
/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c: In function ‘bf0’:
/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c:78:43: error: expected iteration declaration or initialization before ‘x’
78 | for (int x = x0_blk0 - 1, xs = 0; x <= x0_blk0 + x0_blk0_size; x += 1, xs += 1)
| ^
/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c:80:45: error: expected iteration declaration or initialization before ‘y’
80 | for (int y = y0_blk0 - 1, ys = 0; y <= y0_blk0 + y0_blk0_size; y += 1, ys += 1)
| ^
/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c:90:39: error: expected iteration declaration or initialization before ‘x’
90 | for (int x = x0_blk0, xs = 0; x <= x0_blk0 + x0_blk0_size - 1; x += 1, xs += 1)
| ^
/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c:92:41: error: expected iteration declaration or initialization before ‘y’
92 | for (int y = y0_blk0, ys = 0; y <= y0_blk0 + y0_blk0_size - 1; y += 1, ys += 1)
| ^
FAILED compiler invocation:gcc -O3 -g -fPIC -Wall -std=c99 -Wno-unused-result -Wno-unused-variable -Wno-unused-but-set-variable -ffast-math -shared -fopenmp /tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c -lm -o /tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.so
Could you open Pull Requests to address these small issues?
I can see that in your logs:
op2 = Operator(Eq(u, u + 1), platform='nvidiaX')
Please try setting
DEVITO_PLATFORM=arm
We are working on a patch to solve issues ASAP.
@fcannini this PR: https://github.com/devitocodes/devito/pull/1515 is supposed to fix the issues with ARM, and is tested on ThunderX2. as shown here: https://github.com/devitocodes/devito/wiki/%5BManual%5D-Installing-Devito-on-ARM.-Tested-on-GW4-Isambard
Can you try it?
Your compilation issue logged as:
/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c: In function ‘bf0’:
/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c:78:43: error: expected iteration declaration or initialization before ‘x’
78 | for (int x = x0_blk0 - 1, xs = 0; x <= x0_blk0 + x0_blk0_size; x += 1, xs += 1)
| ^
/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c:80:45: error: expected iteration declaration or initialization before ‘y’
80 | for (int y = y0_blk0 - 1, ys = 0; y <= y0_blk0 + y0_blk0_size; y += 1, ys += 1)
| ^
/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c:90:39: error: expected iteration declaration or initialization before ‘x’
90 | for (int x = x0_blk0, xs = 0; x <= x0_blk0 + x0_blk0_size - 1; x += 1, xs += 1)
| ^
/tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c:92:41: error: expected iteration declaration or initialization before ‘y’
92 | for (int y = y0_blk0, ys = 0; y <= y0_blk0 + y0_blk0_size - 1; y += 1, ys += 1)
| ^
FAILED compiler invocation:gcc -O3 -g -fPIC -Wall -std=c99 -Wno-unused-result -Wno-unused-variable -Wno-unused-but-set-variable -ffast-math -shared -fopenmp /tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.c -lm -o /tmp/devito-jitcache-uid927800007/e20a23bf10888f2e802213becebab45f3cf8d1a5.so
is probably happening due to faulty core autodetection. (It happened to me as well.) The PR mentioned above is supposed to fix it.
Let us know if this works for you.
@georgebisbas When running with DEVITO_PLATFORM=arm
the error messages are the same as my first comment.
PR #1515 is not supposed to fix all tests, as code generation is kinda different for AMR kernels (no denormals flushing etc...) but you should be able to run al operators and most of the tests. Could you have a look at it? --George
@georgebisbas I've applied #1515 as a patch but still the same tests are failing. test_dse.py::TestAliases::test_full_shape test_dse.py::TestAliases::test_contracted_shape test_dse.py::TestAliases::test_uncontracted_shape test_dse.py::TestAliases::test_full_shape_w_subdims test_dse.py::TestAliases::test_mixed_shapes
Hi again @fcannini , may I ask how you run these tests? (reproducibility instructions?). I am using a ThunderX2 as well.
So, just to note that at this moment we are not expecting all the tests to pass cause we do not have yet an ARM-CI and the code-generation is kinda different from the typical CPU code generation.
However, the above tests pass for me! :
brx-gbisbas@xcimom2:~/pbs.77294.xci00.x8z/devito> python3 -m pytest tests/test_dse.py::TestAliases::test_full_shape
======================================================= test session starts =======================================================
platform linux -- Python 3.6.5, pytest-6.1.2, py-1.9.0, pluggy-0.13.0
rootdir: /lustre/home/brx-gbisbas/pbs.77294.xci00.x8z/devito, configfile: setup.cfg
plugins: forked-1.3.0, nbval-0.9.3, xdist-2.1.0, cov-2.8.1
collected 2 items
tests/test_dse.py .. [100%]
======================================================== warnings summary =========================================================
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
/opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216, got 192
return f(*args, **kwds)
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
/opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
return f(*args, **kwds)
-- Docs: https://docs.pytest.org/en/stable/warnings.html
====================================================== slowest 20 durations =======================================================
2.53s call tests/test_dse.py::TestAliases::test_full_shape[False]
1.99s call tests/test_dse.py::TestAliases::test_full_shape[True]
(4 durations < 0.005s hidden. Use -vv to show these durations.)
================================================== 2 passed, 3 warnings in 8.81s ==================================================
brx-gbisbas@xcimom2:~/pbs.77294.xci00.x8z/devito> python3 -m pytest tests/test_dse.py::TestAliases::test_contracted_shape
======================================================= test session starts =======================================================
platform linux -- Python 3.6.5, pytest-6.1.2, py-1.9.0, pluggy-0.13.0
rootdir: /lustre/home/brx-gbisbas/pbs.77294.xci00.x8z/devito, configfile: setup.cfg
plugins: forked-1.3.0, nbval-0.9.3, xdist-2.1.0, cov-2.8.1
collected 2 items
tests/test_dse.py .. [100%]
======================================================== warnings summary =========================================================
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
/opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216, got 192
return f(*args, **kwds)
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
/opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
return f(*args, **kwds)
-- Docs: https://docs.pytest.org/en/stable/warnings.html
====================================================== slowest 20 durations =======================================================
2.16s call tests/test_dse.py::TestAliases::test_contracted_shape[False]
1.93s call tests/test_dse.py::TestAliases::test_contracted_shape[True]
(4 durations < 0.005s hidden. Use -vv to show these durations.)
================================================== 2 passed, 3 warnings in 5.26s ==================================================
brx-gbisbas@xcimom2:~/pbs.77294.xci00.x8z/devito> python3 -m pytest tests/test_dse.py::TestAliases::test_mixed_shapes
======================================================= test session starts =======================================================
platform linux -- Python 3.6.5, pytest-6.1.2, py-1.9.0, pluggy-0.13.0
rootdir: /lustre/home/brx-gbisbas/pbs.77294.xci00.x8z/devito, configfile: setup.cfg
plugins: forked-1.3.0, nbval-0.9.3, xdist-2.1.0, cov-2.8.1
collected 2 items
tests/test_dse.py .. [100%]
======================================================== warnings summary =========================================================
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
/opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216, got 192
return f(*args, **kwds)
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
/opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
return f(*args, **kwds)
-- Docs: https://docs.pytest.org/en/stable/warnings.html
====================================================== slowest 20 durations =======================================================
2.72s call tests/test_dse.py::TestAliases::test_mixed_shapes[False]
2.59s call tests/test_dse.py::TestAliases::test_mixed_shapes[True]
(4 durations < 0.005s hidden. Use -vv to show these durations.)
================================================== 2 passed, 3 warnings in 6.61s ==================================================
brx-gbisbas@xcimom2:~/pbs.77294.xci00.x8z/devito> python3 -m pytest tests/test_dse.py::TestAliases::test_full_shape_w_subdims
======================================================= test session starts =======================================================
platform linux -- Python 3.6.5, pytest-6.1.2, py-1.9.0, pluggy-0.13.0
rootdir: /lustre/home/brx-gbisbas/pbs.77294.xci00.x8z/devito, configfile: setup.cfg
plugins: forked-1.3.0, nbval-0.9.3, xdist-2.1.0, cov-2.8.1
collected 2 items
tests/test_dse.py .. [100%]
======================================================== warnings summary =========================================================
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
/opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216, got 192
return f(*args, **kwds)
../../../../../opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219
/opt/python/3.6.5.6/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
return f(*args, **kwds)
-- Docs: https://docs.pytest.org/en/stable/warnings.html
====================================================== slowest 20 durations =======================================================
2.53s call tests/test_dse.py::TestAliases::test_full_shape_w_subdims[False]
2.40s call tests/test_dse.py::TestAliases::test_full_shape_w_subdims[True]
(4 durations < 0.005s hidden. Use -vv to show these durations.)
================================================== 2 passed, 3 warnings in 6.00s ==================================================
Nevertheless there are quite a few failing tests with this patch. In order to see all tests failing and not just 5 (default maxfail value) try running with pytest --maxfail 100
(100 can be a whatever big value....)
Those tests are affected by the code generation procedure followed and are mostly expected to fail. May I ask what is your main purpose for this?
Cause I guess, if you would like to run Devito operators you should be just fine to continue your work as that aspect seems to be just fine.
Let me know about reproducibility instructions and how you test please. Thanks a lot.
Those tests are affected by the code generation procedure followed and are mostly expected to fail. May I ask what is your main purpose for this?
I'm not an user of devito, I'm "the IT guy" tasked to install it on behalf of a client. I'm using the tests as a sort of checklist to see if everything is okay so I can deliver it to the users.
Cause I guess, if you would like to run Devito operators you should be just fine to continue your work as that aspect seems to be just fine.
Am I correct in understanding that these failures aren't that big of a deal?
Let me know about reproducibility instructions and how you test please. Thanks a lot.
HI again,
Not all Devito tests are expected to run on ARM. As far as I can tell, Devito is installed correctly and should work fine. Should your users bump into any issues, please tell them to get in touch with us directly (e.g., on our Slack workspace) .
You should be able to run wave-propagators in Devito without problems. e.g. on ARM Thunder X2 I see that wave-propagator operators are running.
@georgebisbas I don't think it merits a PR, but this gist should be helpful (somehow) https://gist.github.com/fcannini/70b66c0bea604d7f62ab9e83ba7b9b0f
HI @fcannini, my apologies for the delay here. This PR : https://github.com/devitocodes/devito/pull/1515/files should have solved quite a few issues and has been useful for us in the last months. Though Devito still doesn't have ARM CI, so support is not guaranteed. I am closing this issue now, let us know if you need any help, either by reopening this issue or by posting in our Slack, and specifically in the #arm channel. Thanks.
Hello there.
I'm having some issues running tests of version 4.2.3 on a Cavium ThunderX2 node running centos 7.7. Compilers are GCC 9.2.0 built by spack.
test_operator.py::TestOperatorArguments::test_platform_compiler_language fails with the following error.
test_dse.py::TestAliases::test_full_shape, test_dse.py::TestAliases::test_contracted_shape, test_dse.py::TestAliases::test_uncontracted_shape, test_dse.py::TestAliases::test_uncontracted_shape_invariants, test_dse.py::TestAliases::test_full_shape_w_subdims, test_dse.py::TestAliases::test_mixed_shapes all fail with this same error (except for minute differences of file paths).
Every other test passes with flying colors.