search

JuliaPy / PythonCall.jl

Python and Julia in harmony.
https://juliapy.github.io/PythonCall.jl/stable/
MIT License
715 stars 61 forks source link

172407 segmentation fault (core dumped) python #470

Open mcamp-ata opened 3 months ago

mcamp-ata commented 3 months ago

Affects: PythonCall / JuliaCall / Both [delete as appropriate]

Describe the bug

When I use UNet.jl from Python using juliacall I get a segmentation fault (core dumped). When I went to reproduce this error with a minimal example it did work once, but then failed a second time. If I had to guess it might be related to the actual data being passed to the Unet model; maybe its spitting out a type that isn't compatible with Python.

This is a minimal example of the error. I activate my environment that has UNet.jl installed in it, then create some random data to simulate an image and pass it to an untrained model. I get the same results with real images and a trained model. 

Python 3.11.7 (main, Dec  4 2023, 18:10:11) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from juliacall import Main as jl
>>> jl.seval("using Pkg")
>>> jl.Pkg.activate("./model")
  Activating project at `~/work-code/sentinal-hub/model`
>>> jl.seval("using UNet")
>>> img_proxy = jl.rand(256,256,3,1)
>>> unet = jl.Unet(3)
>>> unet(img_proxy)
┌ Warning: Slow fallback implementation invoked for conv!  You probably don't want this; check your datatypes.
│   yT = Float64
│   T1 = Float64
│   T2 = Float32
└ @ NNlib ~/.config/julia/packages/NNlib/Fg3DQ/src/conv.jl:192
┌ Warning: Slow fallback implementation invoked for ∇conv_data!  You probably don't want this; check your datatypes.
│   yT = Float64
│   T1 = Float64
│   T2 = Float32
└ @ NNlib ~/.config/julia/packages/NNlib/Fg3DQ/src/conv.jl:233
Julia:
256×256×3×1 Array{Float64, 4}:
[:, :, 1, 1] =
 -0.0010724    -0.00112552   -0.00113706   …  -0.00103519   -0.000839464
 -0.000795291  -0.00161727   -0.00204406      -0.0009993    -0.000439908
 -0.000780577  -0.00116744   -0.00134499      -0.00115543   -0.000573814
 -0.000860885  -0.00116414   -0.000965194     -0.00140911   -0.000761458
 -0.000755788  -0.00132476   -0.00171172      -0.00114426   -0.000614211
 -0.000820267  -0.00138928   -0.00182287   …  -0.00135342   -0.000624007
 -0.00142297   -0.0017854    -0.00136742      -0.00158695   -0.000791874
 -0.00044614   -0.00124075   -0.00200514      -0.00151648   -0.00105216
 -0.00127456   -0.00144327   -0.00143971      -0.00160615   -0.000831712
  ⋮                                        ⋱                 ⋮
 -0.00081029   -0.00123124   -0.00143105      -0.00146172   -0.00102908
 -0.00142603   -0.00151539   -0.00138619      -0.00166838   -0.0007978
 -0.000832417  -0.0018811    -0.0018118       -0.00110241   -0.000492823
 -0.0010199    -0.00116831   -0.00139769   …  -0.00111267   -0.000343543
 -0.00121636   -0.00164644   -0.0013786       -0.0010909    -0.000885556
 -0.000991607  -0.00133452   -0.00151132      -0.00168399   -0.000749311
 -0.000765725  -0.00120822   -0.00101517      -0.00143129   -0.000483814
 -0.000692864  -0.000724345  -0.00136182      -0.00145267   -0.000722094
 -0.000375997  -0.000676651  -0.00050234   …  -0.000409303   1.38022e-5

[:, :, 2, 1] =
 -0.000460067  -0.000107008   5.77905e-6   …  -0.000240988   0.000301157
 -0.000478966  -0.000925048  -0.000427395     -0.000241288  -0.000264133
 -0.000360445  -2.95737e-5   -0.000434373     -0.000289934  -6.69133e-5
 -0.000216115  -0.000247443  -0.000172126     -0.000211383  -0.000136124
 -0.000232735  -0.000382114  -0.000460042     -8.31166e-5   -0.000159839
 -0.000589542  -8.42927e-5   -0.000223337  …  -0.000574704  -5.28923e-5
 -0.000425194  -0.000388782  -0.000240721     -0.000288628  -0.000253867
 -0.000480769  -0.000710803  -0.000844951     -0.000207783  -0.000235723
 -0.000341701   0.000267551  -0.000264922      3.98389e-6   -0.000140127
  ⋮                                        ⋱                 ⋮
 -0.0003459    -2.07316e-5   -0.000133522     -0.000437807  -0.000310041
 -0.00046411   -0.000290278  -7.01407e-5      -0.000545766  -0.000203747
 -0.000626812  -0.000646226  -0.000518441     -0.000184424  -0.000245369
 -0.000348504  -0.00016867   -0.000313526  …  -0.000363552  -0.000242687
 -0.000545295  -0.000355292  -0.000290152     -0.000299346   0.00015746
 -0.00025227   -0.000617288  -0.000418469     -0.000420933  -5.88442e-5
 -0.000257633  -8.63061e-5   -0.000428485     -8.45088e-5   -0.000438236
 -0.000102137  -0.000326709  -0.000400668     -0.000355772  -1.43647e-5
 -0.000100777  -0.00015405   -7.97384e-5   …  -0.000520814  -0.000936311

[:, :, 3, 1] =
 -0.000548159  -0.000477585  -0.000481873  …  -0.000418627  -0.000233547
 -0.000336213  -0.000695811  -0.000715167     -0.000361649  -0.000291876
 -0.000255482  -0.000366544  -0.000544169     -0.000500734  -0.0003229
 -0.000300283  -0.000279591  -0.000585925     -0.000335869  -0.000303898
 -0.000252053  -0.000579463  -0.000467105     -0.000513136  -0.00031526
 -0.000468617  -0.000659257  -0.000710513  …  -0.00054457   -0.000389497
 -0.000372169  -0.000483613  -0.000773947     -0.000423746  -0.00058391
 -0.000217254  -0.000644031  -0.000634734     -0.000646502  -0.000405915
 -0.00033467   -0.000407474  -0.000578466     -0.000392831  -0.000199668
  ⋮                                        ⋱                 ⋮
 -0.000340643  -0.000672335  -0.000457055     -0.000829627  -0.000533199
 -0.000442569  -0.000660255  -0.000655409     -0.000518397  -0.000289253
 -0.000479983  -0.000490807  -0.000667401     -0.000384335  -0.000380235
 -0.00028761   -0.000664416  -0.000677983  …  -0.000415594  -0.000365336
 -0.000468542  -0.000480694  -0.000676793     -0.000697129  -0.000421928
 -0.000230119  -0.000454513  -0.000582444     -0.000374222  -0.00037701
 -0.000217566  -0.000276869  -0.000450451     -0.000470439  -0.000380189
 -5.60605e-5   -0.000273944  -0.000347952     -6.15815e-5   -0.000395992
  2.86524e-5   -3.08762e-5   -5.8798e-5    …  -0.000350873  -0.000154495
>>> unet(img_proxy)
Julia:
256×256×3×1 Array{Float64, 4}:
[:, :, 1, 1] =
 -0.0010724    -0.00112552   -0.00113706   …  -0.00103519   -0.000839464
 -0.000795291  -0.00161727   -0.00204406      -0.0009993    -0.000439908
 -0.000780577  -0.00116744   -0.00134499      -0.00115543   -0.000573814
 -0.000860885  -0.00116414   -0.000965194     -0.00140911   -0.000761458
 -0.000755788  -0.00132476   -0.00171172      -0.00114426   -0.000614211
 -0.000820267  -0.00138928   -0.00182287   …  -0.00135342   -0.000624007
 -0.00142297   -0.0017854    -0.00136742      -0.00158695   -0.000791874
 -0.00044614   -0.00124075   -0.00200514      -0.00151648   -0.00105216
 -0.00127456   -0.00144327   -0.00143971      -0.00160615   -0.000831712
  ⋮                                        ⋱                 ⋮
 -0.00081029   -0.00123124   -0.00143105      -0.00146172   -0.00102908
 -0.00142603   -0.00151539   -0.00138619      -0.00166838   -0.0007978
 -0.000832417  -0.0018811    -0.0018118       -0.00110241   -0.000492823
 -0.0010199    -0.00116831   -0.00139769   …  -0.00111267   -0.000343543
 -0.00121636   -0.00164644   -0.0013786       -0.0010909    -0.000885556
 -0.000991607  -0.00133452   -0.00151132      -0.00168399   -0.000749311
 -0.000765725  -0.00120822   -0.00101517      -0.00143129   -0.000483814
 -0.000692864  -0.000724345  -0.00136182      -0.00145267   -0.000722094
 -0.000375997  -0.000676651  -0.00050234   …  -0.000409303   1.38022e-5

[:, :, 2, 1] =
 -0.000460067  -0.000107008   5.77905e-6   …  -0.000240988   0.000301157
 -0.000478966  -0.000925048  -0.000427395     -0.000241288  -0.000264133
 -0.000360445  -2.95737e-5   -0.000434373     -0.000289934  -6.69133e-5
 -0.000216115  -0.000247443  -0.000172126     -0.000211383  -0.000136124
 -0.000232735  -0.000382114  -0.000460042     -8.31166e-5   -0.000159839
 -0.000589542  -8.42927e-5   -0.000223337  …  -0.000574704  -5.28923e-5
 -0.000425194  -0.000388782  -0.000240721     -0.000288628  -0.000253867
 -0.000480769  -0.000710803  -0.000844951     -0.000207783  -0.000235723
 -0.000341701   0.000267551  -0.000264922      3.98389e-6   -0.000140127
  ⋮                                        ⋱                 ⋮
 -0.0003459    -2.07316e-5   -0.000133522     -0.000437807  -0.000310041
 -0.00046411   -0.000290278  -7.01407e-5      -0.000545766  -0.000203747
 -0.000626812  -0.000646226  -0.000518441     -0.000184424  -0.000245369
 -0.000348504  -0.00016867   -0.000313526  …  -0.000363552  -0.000242687
 -0.000545295  -0.000355292  -0.000290152     -0.000299346   0.00015746
 -0.00025227   -0.000617288  -0.000418469     -0.000420933  -5.88442e-5
 -0.000257633  -8.63061e-5   -0.000428485     -8.45088e-5   -0.000438236
 -0.000102137  -0.000326709  -0.000400668     -0.000355772  -1.43647e-5
 -0.000100777  -0.00015405   -7.97384e-5   …  -0.000520814  -0.000936311

[:, :, 3, 1] =
 -0.000548159  -0.000477585  -0.000481873  …  -0.000418627  -0.000233547
 -0.000336213  -0.000695811  -0.000715167     -0.000361649  -0.000291876
 -0.000255482  -0.000366544  -0.000544169     -0.000500734  -0.0003229
 -0.000300283  -0.000279591  -0.000585925     -0.000335869  -0.000303898
 -0.000252053  -0.000579463  -0.000467105     -0.000513136  -0.00031526
 -0.000468617  -0.000659257  -0.000710513  …  -0.00054457   -0.000389497
 -0.000372169  -0.000483613  -0.000773947     -0.000423746  -0.00058391
 -0.000217254  -0.000644031  -0.000634734     -0.000646502  -0.000405915
 -0.00033467   -0.000407474  -0.000578466     -0.000392831  -0.000199668
  ⋮                                        ⋱                 ⋮
 -0.000340643  -0.000672335  -0.000457055     -0.000829627  -0.000533199
 -0.000442569  -0.000660255  -0.000655409     -0.000518397  -0.000289253
 -0.000479983  -0.000490807  -0.000667401     -0.000384335  -0.000380235
 -0.00028761   -0.000664416  -0.000677983  …  -0.000415594  -0.000365336
 -0.000468542  -0.000480694  -0.000676793     -0.000697129  -0.000421928
 -0.000230119  -0.000454513  -0.000582444     -0.000374222  -0.00037701
 -0.000217566  -0.000276869  -0.000450451     -0.000470439  -0.000380189
 -5.60605e-5   -0.000273944  -0.000347952     -6.15815e-5   -0.000395992
  2.86524e-5   -3.08762e-5   -5.8798e-5    …  -0.000350873  -0.000154495
>>> img_proxy = jl.rand(256,256,3,1)
>>> unet(img_proxy)
[1]    172407 segmentation fault (core dumped)  python

Your system I am running NixOS 23.11 and below is the Poetry environment that I am using with juliacall. Julia is version 1.10.0. 

───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: pyproject.toml
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ [tool.poetry]
   2   │ name = "scientific_test"
   3   │ version = "0.1.0"
   4   │ description = ""
   5   │ authors = ["Your Name <you@example.com>"]
   6   │ 
   7   │ [tool.poetry.dependencies]
   8   │ python = "^3.10"
   9   │ matplotlib = "*"
  10   │ pandas = "*"
  11   │ requests = "^2.31.0"
  12   │ pillow = "^10.1.0"
  13   │ pyjulia = "^0.0.6"
  14   │ juliacall = "^0.9.14"
  15   │ julia = "^0.6.1"
  16   │ mlflow = "*"
  17   │ setuptools = "^69.0.3"
  18   │ setuptools-scm = "^8.0.4"
  19   │ boto3 = "^1.34.33"
  20   │ pip = "^23.3.2"
  21   │ virtualenv = "^20.25.0"
  22   │ gunicorn = "^21.2.0"
  23   │ cloudpickle = "^3.0.0"
  24   │ 
  25   │ [build-system]
  26   │ requires = ["poetry-core>=1.1"]
  27   │ build-backend = "poetry.core.masonry.api"
  28   │ 
───────┴──────────────────────────────────────────────────
Status `~/work-code/sentinal-hub/model/Project.toml`
  [fbe9abb3] AWS v1.90.4
  [1c724243] AWSS3 v0.11.2
⌅ [79e6a3ab] Adapt v3.7.2
  [c7e460c6] ArgParse v1.1.5
  [02898b10] Augmentor v0.6.6
  [fbb218c0] BSON v0.3.9
  [6e4b80f9] BenchmarkTools v1.5.0
⌅ [052768ef] CUDA v3.13.1
  [3da002f7] ColorTypes v0.11.4
  [992eb4ea] CondaPkg v0.2.22
  [864edb3b] DataStructures v0.18.18
  [44557152] DateFormats v0.1.18
  [5789e2e9] FileIO v1.16.2
⌅ [587475ba] Flux v0.12.10
⌃ [28b8d3ca] GR v0.72.8
  [4c0ca9eb] Gtk v1.3.0
  [cd3eb016] HTTP v1.10.3
  [7073ff75] IJulia v1.24.2
⌅ [a09fc81d] ImageCore v0.8.22
  [4381153b] ImageDraw v0.2.6
⌅ [6a3955dd] ImageFiltering v0.6.21
⌅ [82e4d734] ImageIO v0.5.9
⌃ [6218d12a] ImageMagick v1.3.0
⌅ [02fcd773] ImageTransformations v0.8.13
⌃ [86fae568] ImageView v0.10.14
⌅ [916415d5] Images v0.24.1
  [033835bb] JLD2 v0.4.46
  [682c06a0] JSON v0.21.4
  [a21d8daa] LabelStudioHelper v1.0.0-DEV `https://gitlab.com/usmcamp0811/labelstudiohelper#main`
  [64a0f543] MLFlowClient v0.4.4 `https://github.com/usmcamp0811/MLFlowClient.jl#main`
  [4281f0d9] Minio v0.2.1
  [e94cdb99] MosaicViews v0.3.4
  [6fe1bfb0] OffsetArrays v1.13.0
  [5fb14364] OhMyREPL v0.5.24
  [91a5bcdd] Plots v1.40.1
  [c3e4b0f8] Pluto v0.19.40
  [92933f4c] ProgressMeter v1.10.0
  [438e738f] PyCall v1.96.4
  [6099a3de] PythonCall v0.9.15
⌅ [2913bbd2] StatsBase v0.33.21
  [5e47fb64] TestImages v1.8.0
  [0d73aaa9] UNet v0.2.1
  [104b5d7c] WebSockets v1.6.0
⌃ [da03df04] gdk_pixbuf_jll v2.42.8+0
  [ade2ca70] Dates
  [4af54fe1] LazyArtifacts
  [8dfed614] Test
Info Packages marked with ⌃ and ⌅ have new versions available. Those with ⌃ may be upgradable, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated`
MilesCranmer commented 3 months ago

Cannot reproduce (I'm on macOS). UNet seems to work fine for me.

Did you start Python with PYTHON_JULIACALL_HANDLE_SIGNALS=yes in the environment variable? Otherwise it can have segfaults due to multithreading (since Python has a global interpreter lock).

cjdoris commented 3 months ago

see here for a bit more context on @MilesCranmer's comment