search

kailaix / ADCME.jl

Automatic Differentiation Library for Computational and Mathematical Engineering
https://kailaix.github.io/ADCME.jl/latest/
MIT License
286 stars 57 forks source link

Build Custom Operators error #76

Open liufeng2317 opened 2 years ago

liufeng2317 commented 2 years ago

hello!   when I trying to run the custom operators example , it occurs some errors as follows,it seems that there are some wrongs happened with the tensorflow but I don't know how to deal with it.   and I am confued with signal (6): Aborted problem.If you can solve this problem, I would appreciate it!

test@P340:/data/test/project/007_inversion/ADCME_test/test_custom$ julia gradtest.jl 
Load library operator (with gradient, multiple outputs = false): /data/test/project/007_inversion/ADCME_test/test_custom/build/libMySparseSolver.so ==> my_sparse_solver
2021-12-29 17:23:18.679176: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.
2021-12-29 17:23:18.700633: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2899885000 Hz
2021-12-29 17:23:18.701170: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4071d80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-12-29 17:23:18.701186: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
OMP: Info #212: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #210: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0-15
OMP: Info #156: KMP_AFFINITY: 16 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 8 cores/pkg x 2 threads/core (8 total cores)
OMP: Info #214: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 0 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 1 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 1 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 2 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 2 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 3 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 3 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 4 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 12 maps to package 0 core 4 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 5 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 13 maps to package 0 core 5 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 6 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 14 maps to package 0 core 6 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 7 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 15 maps to package 0 core 7 thread 1 
OMP: Info #250: KMP_AFFINITY: pid 22180 tid 22180 thread 0 bound to OS proc set 0
2021-12-29 17:23:18.702430: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2021-12-29 17:23:18.741479: F tensorflow/core/framework/tensor_shape.cc:324] Check failed: size >= 0 (-1 vs. 0)

signal (6): Aborted
in expression starting at /data/test/project/007_inversion/ADCME_test/test_custom/gradtest.jl:23
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_ZN10tensorflow8internal15LogMessageFatalD1Ev at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow15TensorShapeBaseINS_11TensorShapeEE6AddDimEx at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/../libtensorflow_framework.so.1 (unknown line)
_ZN10tensorflow15TensorShapeBaseINS_11TensorShapeEE8InitDimsEN4absl4SpanIKxEE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/../libtensorflow_framework.so.1 (unknown line)
_ZN16MySparseSolverOp7ComputeEPN10tensorflow15OpKernelContextE at /data/test/project/007_inversion/ADCME_test/test_custom/build/libMySparseSolver.so (unknown line)
_ZN10tensorflow8grappler12EvaluateNodeERKNS_7NodeDefERKN4absl13InlinedVectorINS_11TensorValueELm4ESaIS6_EEEPNS_10DeviceBaseEPNS_11ResourceMgrEPS8_ at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZNK10tensorflow8grappler15ConstantFolding12EvaluateNodeERKNS_7NodeDefERKN4absl13InlinedVectorINS_11TensorValueELm4ESaIS7_EEEPS9_ at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow8grappler15ConstantFolding19EvaluateOneFoldableERKNS_7NodeDefEPSt6vectorIS2_SaIS2_EEPb at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow8grappler15ConstantFolding8FoldNodeEPNS_7NodeDefEPNS_8GraphDefEPb at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow8grappler15ConstantFolding9FoldGraphERKNS0_15GraphPropertiesEPNS_8GraphDefEPN4absl13flat_hash_setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS7_18container_internal10StringHashENSF_12StringHashEq2EqESaISE_EEE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow8grappler15ConstantFolding19RunOptimizationPassEPNS0_7ClusterERKNS0_12GrapplerItemEPNS_8GraphDefE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow8grappler15ConstantFolding8OptimizeEPNS0_7ClusterERKNS0_12GrapplerItemEPNS_8GraphDefE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow8grappler13MetaOptimizer12RunOptimizerEPNS0_14GraphOptimizerEPNS0_7ClusterEPNS0_12GrapplerItemEPNS_8GraphDefEPNS1_23GraphOptimizationResultE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow8grappler13MetaOptimizer13OptimizeGraphEPNS0_7ClusterERKNS0_12GrapplerItemEPNS_8GraphDefE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow8grappler13MetaOptimizer8OptimizeEPNS0_7ClusterERKNS0_12GrapplerItemEPNS_8GraphDefE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow8grappler16RunMetaOptimizerERKNS0_12GrapplerItemERKNS_11ConfigProtoEPNS_10DeviceBaseEPNS0_7ClusterEPNS_8GraphDefE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow19GraphExecutionState13OptimizeGraphERKNS_17BuildGraphOptionsEPSt10unique_ptrINS_5GraphESt14default_deleteIS5_EEPS4_INS_25FunctionLibraryDefinitionES6_ISA_EE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow19GraphExecutionState10BuildGraphERKNS_17BuildGraphOptionsEPSt10unique_ptrINS_11ClientGraphESt14default_deleteIS5_EE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow13DirectSession12CreateGraphsERKNS_17BuildGraphOptionsEPSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt10unique_ptrINS_5GraphESt14default_deleteISC_EESt4hashISA_ESt8equal_toISA_ESaISt4pairIKSA_SF_EEEPSB_INS_25FunctionLibraryDefinitionESD_ISQ_EEPNS0_12RunStateArgsEPN4absl13InlinedVectorINS_8DataTypeELm4ESaISY_EEES11_Px at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow13DirectSession15CreateExecutorsERKNS_15CallableOptionsEPSt10unique_ptrINS0_16ExecutorsAndKeysESt14default_deleteIS5_EEPS4_INS0_12FunctionInfoES6_ISA_EEPNS0_12RunStateArgsE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow13DirectSession20GetOrCreateExecutorsEN4absl4SpanIKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEESA_SA_PPNS0_16ExecutorsAndKeysEPNS0_12RunStateArgsE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow13DirectSession3RunERKNS_10RunOptionsERKSt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_6TensorEESaISD_EERKS4_ISB_SaISB_EESL_PS4_ISC_SaISC_EEPNS_11RunMetadataE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow10SessionRef3RunERKNS_10RunOptionsERKSt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_6TensorEESaISD_EERKS4_ISB_SaISB_EESL_PS4_ISC_SaISC_EEPNS_11RunMetadataE at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZL13TF_Run_HelperPN10tensorflow7SessionEPKcPK9TF_BufferRKSt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_6TensorEESaISG_EERKS7_ISE_SaISE_EEPP9TF_TensorSO_PS4_P9TF_Status.constprop.637 at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
TF_SessionRun at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow28TF_SessionRun_wrapper_helperEP10TF_SessionPKcPK9TF_BufferRKSt6vectorI9TF_OutputSaIS8_EERKS7_IP7_objectSaISE_EESC_RKS7_IP12TF_OperationSaISK_EEPS4_P9TF_StatusPSG_ at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_ZN10tensorflow21TF_SessionRun_wrapperEP10TF_SessionPK9TF_BufferRKSt6vectorI9TF_OutputSaIS6_EERKS5_IP7_objectSaISC_EESA_RKS5_IP12TF_OperationSaISI_EEPS2_P9TF_StatusPSE_ at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_wrap_TF_SessionRun_wrapper at /home/node/.julia/adcme/lib/python3.7/site-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so (unknown line)
_PyMethodDef_RawFastCallKeywords at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyCFunction_FastCallKeywords at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1542 at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1542 at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallKeywords at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1542 at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallKeywords at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1542 at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1542 at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_Call_Prepend at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
PyObject_Call at /home/node/.julia/adcme/lib/libpython3.7m.so.1.0 (unknown line)
macro expansion at /home/node/.julia/packages/PyCall/3fwVL/src/exception.jl:95 [inlined]
#107 at /home/node/.julia/packages/PyCall/3fwVL/src/pyfncall.jl:43 [inlined]
disable_sigint at ./c.jl:458 [inlined]
__pycall! at /home/node/.julia/packages/PyCall/3fwVL/src/pyfncall.jl:42 [inlined]
_pycall! at /home/node/.julia/packages/PyCall/3fwVL/src/pyfncall.jl:29
_pycall! at /home/node/.julia/packages/PyCall/3fwVL/src/pyfncall.jl:11
unknown function (ip: 0x7f817c4bbc33)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
#_#114 at /home/node/.julia/packages/PyCall/3fwVL/src/pyfncall.jl:86
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:670
PyObject at /home/node/.julia/packages/PyCall/3fwVL/src/pyfncall.jl:86
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
#run#67 at /home/node/.julia/packages/ADCME/hCmEo/src/run.jl:73
run at /home/node/.julia/packages/ADCME/hCmEo/src/run.jl:69
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:562
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
include_string at ./loading.jl:1116
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
_include at ./loading.jl:1170
include at ./Base.jl:386
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
exec_options at ./client.jl:285
_start at ./client.jl:485
jfptr__start_43689.clone_1 at /home/node/Downloads/julia-1.6.3/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
true_main at /buildworker/worker/package_linux64/build/src/jlapi.c:560
repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702
main at julia (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4007d8)
Allocations: 9370211 (Pool: 9366630; Big: 3581); GC: 12
Aborted (core dumped)