search

uber / petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Apache License 2.0
1.8k stars 284 forks source link

Segmentation fault importing pytorch Dataloader #494

Closed sonNeturo closed 4 years ago

sonNeturo commented 4 years ago

Hello,

Importing the pytorch dataloader causes a segmentation fault and crushes. This doesn't happen locally, but only when I run a simple import in on google dataproc.

import faulthandler
faulthandler.enable()

from petastorm.pytorch import DataLoader

I get:

Waiting for job output...
Fatal Python error: Segmentation fault

Current thread 0x00007f28e9495700 (most recent call first):
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 922 in create_module
  File "<frozen importlib._bootstrap>", line 560 in module_from_spec
  File "<frozen importlib._bootstrap>", line 648 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 950 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load
  File "/opt/conda/miniconda3/envs/deep-ctr/lib/python3.6/site-packages/torch/__init__.py", line 81 in <module>
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 655 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 950 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 936 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 936 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 936 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load
  File "/opt/conda/miniconda3/envs/deep-ctr/lib/python3.6/site-packages/petastorm/pytorch.py", line 22 in <module>
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 655 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 950 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load

Petastorm version: 0.8.2 Pytorch version: 1.4.0

Thanks in advance for the help

sonNeturo commented 4 years ago

ok, I narrowed it down to importing pyarrow then pytorch:

import faulthandler
faulthandler.enable()
import pyarrow
import torch
Fatal Python error: Segmentation fault

Current thread 0x00007f7c49fce700 (most recent call first):
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 922 in create_module
  File "<frozen importlib._bootstrap>", line 560 in module_from_spec
  File "<frozen importlib._bootstrap>", line 648 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 950 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load
  File "/opt/conda/miniconda3/envs/deep-ctr/lib/python3.6/site-packages/torch/__init__.py", line 81 in <module>
  File "<frozen importlib._bootstrap>", line 205 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 655 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 950 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 961 in _find_and_load
  File "/tmp/0fbad23d7fa345a1b8b0b569aec343d6/draft.py", line 8 in <module>
sonNeturo commented 4 years ago

Fixed the issue by installing pyarrow and pytorch form conda