search

gramineproject / graphene

Graphene / Graphene-SGX - a library OS for Linux multi-process applications, with Intel SGX support
https://grapheneproject.io
GNU Lesser General Public License v3.0
769 stars 260 forks source link

PPML example with SGX Error #2630

Closed adityapribadi3 closed 3 years ago

adityapribadi3 commented 3 years ago

Hi so I follow this guide https://graphene.readthedocs.io/en/latest/tutorials/pytorch/index.html#executing-pytorch-with-graphene-in-sgx-enclave I have successfully run "make SGX=1" however when I run "graphene-sgx ./pytorch pytorchexample.py" it throws me this error

error: Using insecure argv source. Graphene will continue application execution, but this configuration must not be used in production! error: Forwarding host environment variables to the app is enabled. Graphene will continue application execution, but this configuration must not be used in production! /home/ubuntu/.local/lib/python3.6/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.) return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode) Traceback (most recent call last): File "pytorchexample.py", line 47, in with open("result.txt", "w") as outfile: FileExistsError: [Errno 17] File exists: 'result.txt' Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook from apport.fileutils import likely_packaged, get_recent_crashes File "/usr/lib/python3/dist-packages/apport/init.py", line 5, in from apport.report import Report File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in import apport.fileutils File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 25, in from apport.packaging_impl import impl as packaging File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in import apt File "/usr/lib/python3/dist-packages/apt/init.py", line 35, in apt_pkg.init_system() apt_pkg.Error: E:Error reading the CPU table

Original exception was: Traceback (most recent call last): File "pytorchexample.py", line 47, in with open("result.txt", "w") as outfile: FileExistsError: [Errno 17] File exists: 'result.txt'

I have tried to "make clean" run remake the file and also remove the "result.txt" and run again but still, it shows me the same error. I have test with he hello world test using SGX and it works fine

dimakuv commented 3 years ago

Strange, the file should have been overwritten, but Python says that it can't overwrite it...

Does anyone from the Graphene team have time to look into this?

mkow commented 3 years ago

@adityapribadi3 please use proper Markdown formatting for issues, the current version is unreadable.

adityapribadi3 commented 3 years ago

Hi @mkow , here is the screenshot

Screen Shot 2021-08-07 at 1 12 03 am
mkow commented 3 years ago

@adityapribadi3: Sorry for being a bit rude, but have you actually read my comment...?

adityapribadi3 commented 3 years ago

Description of the problem

PPML example with SGX Error when executing

Steps to reproduce

This guide https://graphene.readthedocs.io/en/latest/tutorials/pytorch/index.html#executing-pytorch-with-graphene-in-sgx-enclave (graphene/example/pytorch) make SGX=1 graphene-sgx ./pytorch pytorchexample.py

Expected results

result.txt

Actual results

ubuntu@SGX:~/graphene/Examples/pytorch$ graphene-sgx ./pytorch pytorchexample.py 
error: Using insecure argv source. Graphene will continue application execution, but this configuration must not be used in production!
error: Forwarding host environment variables to the app is enabled. Graphene will continue application execution, but this configuration must not be used in production!
/home/ubuntu/.local/lib/python3.6/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Traceback (most recent call last):
  File "pytorchexample.py", line 47, in <module>
    with open("result.txt", "w") as outfile:
PermissionError: [Errno 13] Permission denied: 'result.txt'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 25, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 35, in <module>
    apt_pkg.init_system()
apt_pkg.Error: E:Error reading the CPU table

Original exception was:
Traceback (most recent call last):
  File "pytorchexample.py", line 47, in <module>
    with open("result.txt", "w") as outfile:
PermissionError: [Errno 13] Permission denied: 'result.txt'
adityapribadi3 commented 3 years ago

Hi @mkow, for the update this error already resolve by changing the script in "pytorchexample.py" when importing the torchvision to "import torchvision.models as models" However now when I run "sudo graphene-sgx ./pytorch pytorchexample.py" I have a different error

ubuntu@VM2:~/graphene/Examples/pytorch$ sudo graphene-sgx ./pytorch pytorchexample.py
error: Using insecure argv source. Graphene will continue application execution, but this configuration must not be used in production!
error: Forwarding host environment variables to the app is enabled. Graphene will continue application execution, but this configuration must not be used in production!
./pytorch: can't open file 'pytorchexample.py': [Errno 13] Permission denied
sahason commented 3 years ago

@mkow PR #2655 seems to address this as well. Please assign it to me.

adityapribadi3 commented 3 years ago

Hi @sahason , is it already fixed?

sahason commented 3 years ago

@adityapribadi3 Yes, please check with this PR #2655.

adityapribadi3 commented 3 years ago

hi sahason, I have try to change the code in "pytorch.manifest.template" and it works but the problem is the example get killed in my machine, I'm using azure DC VM with 16gb RAM, if there a way to limit the RAM so the application is not killed?

ubuntu@VM3:~/graphene/Examples/pytorch$ sudo graphene-sgx ./pytorch ./pytorchexample.py
Killed
sahason commented 3 years ago

@adityapribadi3 I am sure about any such mechanism. @mkow Can you please comment here? One suggestion here is to increase the size of RAM/EPC.

adityapribadi3 commented 3 years ago

I'm using Azure VM with 32Gb RAM and still the process get killed

mkow commented 3 years ago

Does this workload really need 16GB of memory?

I guess the reason you get OOM-ed is that PyTorch is spawning some children, each of which also taking 16GB. And you probably don't have any swap mounted (or just a small one), so the OOM killer kills you right after you cross the 16/32 GB boundary.

adityapribadi3 commented 3 years ago

oh ok then thx, so the only way is upgrading the RAM right?