Closed naveenguptaiitr closed 1 year ago
ar4
commented
1 year ago Hi Naveen,
Thank you for writing.
The only errors that I see in that output seem to be ones like "Could not open output file '/localscratch/1139367/tmpxft_00001ff3_0000000a'". The compiler appears to want to store temporary files in the /localscratch directory and my guess is that it is unable to. Is it possible that it does not have permission to write to that location, or might /localscratch be full?
ar4
commented
1 year ago Hi Naveen,
You might like to try setting the TMPDIR environment variable to a location that nvcc will definitely have write access to (such as a directory you create in your home directory), as the NVCC compiler will apparently then use that location for its temporary files. So you could try "mkdir /home/naveengupta/nvcctmp" and then "export TMPDIR=/home/naveengupta/nvcctmp; python -c 'import deepwave'" (assuming you are using bash). Does that work?
naveenguptaiitr
commented
1 year ago Hi Alan,
I tried this command and still getting the same error. Could you please suggest if you have any insights?
XuVV
commented
1 year ago Hi, Alan. Thanks for the deepwave. I also have some problem installing the newest version of deepwave. I used the command " conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=10.2 -c pytorch" to install pytorch and then used "pip install deepwave" . But when I tried to "import deep wave", I have the error below. So could you please tell me which version of pytorch you used?:
Traceback (most recent call last): File "/nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1808, in _run_ninja_build subprocess.run( File "/nas/home/wxu/miniconda3/envs/lsrtmdip/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/nas/home/wxu/lsrtmdip/lsrtm_test_Marmousi.py", line 4, in
Process finished with exit code 1
ar4
commented
1 year ago @naveenguptaiitr That is a pity. Can you post the new error message? If it is indeed exactly the same - still trying to write to /localscratch - then it seems that NVCC is not using (or receiving) the updated TMPDIR environment variable. If it is using the new TMPDIR and is still unable to write then there seems to be a different problem. Can you also report what the output is when you run the command "touch $TMPDIR/naveentest" on the system that you are trying to use Deepwave on?
naveenguptaiitr
commented
1 year ago I am able to fix the issue with the import using the commands you shared first. The problem was with the updating the TMPDIR variable. Thank you very much for your suggestion and the import issue is fixed.
However, when I am running the FWI inversion on marmousi data, I am getting Location not within the model error. Could you share your insights on this? Thank you again for quickly helping out with the issue.
ar4
commented
1 year ago I am glad to hear that you are now able to use Deepwave.
Are you able to show the code you are using that results in an error? Location not within the model means that one or more of the source or receiver locations are outside the velocity model.
naveenguptaiitr
commented
1 year ago Hi Alan, I was trying to figure out the issue myself and try to understand it, however, I am unable to solve it. I am the using the following code for predicting amplitude for a given velocity model. Please share how to fix the problem. Thank you very much for helping out.!
#################################################### from pysit import * from pysit.gallery import marmousi2
import torch import torch.nn as nn import deepwave from deepwave import scalar
import numpy as np import matplotlib.pyplot as plt
device = torch.device('cuda')
vp_true, vp0, m, d = marmousi2(origin=(7500, 355), size=(1910, 950), pixel_scale="small", initial_model_style="gradient", initial_config={'min':1500.0,'max':3000.0})
vp_true = torch.tensor(vp_true)
dx = m.deltas[0] nz, ny = vp_true.shape num_dims = 2
num_shots = 20
num_sources_per_shot = 1 source_spacing = ny*dx / num_shots #96.0
num_receivers_per_shot = 96 receiver_spacing = ny*dx / num_receivers_per_shot #20.0
peak_freq = 15 dt = 0.004 # 4ms
n_seconds = 4 nt = int(n_seconds / dt)
x_s = torch.zeros(num_shots, num_sources_per_shot, num_dims) x_s[:, 0, 1] = torch.arange(num_shots).float() * source_spacing
x_r = torch.zeros(num_shots, num_receivers_per_shot, num_dims) x_r[0, :, 1] = torch.arange(num_receivers_per_shot).float() * receiver_spacing x_r[:, :, 1] = x_r[0, :, 1].repeat(num_shots, 1)
source_amplitudes = (deepwave.wavelets.ricker(peak_freq, nt, dt, 1/peak_freq).repeat(num_shots, num_sources_per_shot,1))
print(source_amplitudes.shape)
receiver_amplitudes_true = scalar(vp_true, dx, dt, source_amplitudes=source_amplitudes, source_locations=x_s, receiver_locations=x_r, accuracy=8,pml_freq=peak_freq)
#################################################################
Error -
RuntimeError Traceback (most recent call last) Cell In[1], line 60 58 print(source_amplitudes.shape) 59 # predict the amplitude on the marmousi velocity model ---> 60 receiver_amplitudes_true = scalar(vp_true, dx, dt, source_amplitudes=source_amplitudes, 61 source_locations=x_s, receiver_locations=x_r, 62 accuracy=8,pml_freq=peak_freq)
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/scalar.py:360, in scalar(v, grid_spacing, dt, source_amplitudes, source_locations, receiver_locations, accuracy, pml_width, pml_freq, max_vel, survey_pad, wavefield_0, wavefield_m1, psiy_m1, psix_m1, zetay_m1, zetax_m1, origin, nt, model_gradient_sampling_interval, freq_taper_frac, time_pad_frac) 125 def scalar(v: Tensor, 126 grid_spacing: Union[int, float, List[float], 127 Tensor], (...) 147 Tensor, Tensor, 148 Tensor]: 149 """Scalar wave propagation (functional interface). 150 151 This function performs forward modelling with the scalar wave equation. (...) 353 354 """ 355 (models, source_amplitudes_l, wavefields, 356 pml_profiles, sources_i_l, receivers_i_l, 357 dy, dx, dt, nt, n_batch, 358 step_ratio, model_gradient_sampling_interval, 359 accuracy, pml_width_list) = \ --> 360 setup_propagator([v], 'scalar', grid_spacing, dt, 361 [wavefield_0, wavefield_m1, psiy_m1, psix_m1, 362 zetay_m1, zetax_m1], 363 [source_amplitudes], 364 [source_locations], [receiver_locations], 365 accuracy, pml_width, pml_freq, max_vel, 366 survey_pad, 367 origin, nt, model_gradient_sampling_interval, 368 freq_taper_frac, time_pad_frac) 369 v = models[0] 370 wfc, wfp, psiy, psix, zetay, zetax = wavefields
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/common.py:72, in setup_propagator(models, prop_type, grid_spacing, dt, wavefields, source_amplitudes, source_locations, receiver_locations, accuracy, pml_width, pml_freq, max_vel, survey_pad, origin, nt, model_gradient_sampling_interval, freq_taper_frac, time_pad_frac) 70 dy, dx = set_dx(grid_spacing) 71 pad = [fd_pad + width for width in pml_width] ---> 72 models, locations = extract_survey( 73 models, 74 source_locations + receiver_locations, 75 survey_pad, wavefields, origin, pml_width 76 ) 77 source_locations = locations[:len(source_locations)] 78 receiver_locations = locations[len(source_locations):]
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/common.py:584, in extract_survey(models, locations, survey_pad, wavefields, origin, pad) 576 def extract_survey(models: List[Tensor], 577 locations: List[Optional[Tensor]], 578 survey_pad: Optional[Union[int, (...) 582 pad: List[int]) -> Tuple[List[Tensor], 583 List[Optional[Tensor]]]: --> 584 check_locations_are_within_model(models[0].shape, locations) 585 if any([wavefield is not None for wavefield in wavefields]): 586 survey_extents = get_survey_extents_from_wavefields( 587 wavefields, origin, pad 588 )
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/common.py:610, in check_locations_are_within_model(model_shape, locations) 608 for dim, model_dim_shape in enumerate(model_shape): 609 if location[..., dim].max() >= model_dim_shape: --> 610 raise RuntimeError("Locations must be within model.")
RuntimeError: Locations must be within model.
ar4
commented
1 year ago Hi Naveen,
The problem is that your velocity model seems to be shaped incorrectly. When I run your code, the output of print(vp_true.shape, nz, ny) is torch.Size([18432, 1]) 18432 1. Your model is thus a 1D vector rather than a 2D model. If I add the line
vp_true = vp_true.reshape(96, 192)then it will have the correct shape.
Furthermore, Deepwave uses cell indexes as source/receiver locations, so you also need to remove the multiplication by dx when setting your locations, so spacings become
source_spacing = ny / num_shots
receiver_spacing = ny / num_receivers_per_shotIf you need to have locations that are not centered on grid cells then you may wish to look at Deepwave's location interpolation example.
After making these changes I was able to run your code, and produced an image of first shot's data using
vmin, vmax = np.percentile(receiver_amplitudes_true[-1][0], [5,95])
plt.imshow(receiver_amplitudes_true[-1][0].T, aspect='auto', vmin=vmin, vmax=vmax)Hi Alan,
Thank you very much for your help.! This is working and am delighted to get this working. Appreciate your help. I had few more questions to ask. Could you please address following questions?
Thank you again for helping out with the issue. :)
ar4
commented
1 year ago Hi Naveen,
That is good news.
The Usage page of the Deepwave documentation describes all of the inputs and outputs. The other outputs relate the the wavefield at the final timestep. One of the reasons that these are provided is so that calls to Deepwave can be chained together, allowing you to continue propagating to later timesteps (as in the checkpointing example).
On its own, Deepwave does not provide a way to propagate with different velocity models simultaneously, but you might be able to get that to work by combining it with something else, such as by using Python's or PyTorch's multiprocessing capabilities to run Deepwave in multiple processes.
naveenguptaiitr
commented
1 year ago Hi Alan,
Thank you very much for sharing your insights. I will check out the documentation to learn more about the deepwave. It is a great library and indeed super helpful for seismic inversion problems. Thank you for helping out.
Hi Alan,
I installed the latest version of deepwave and the installation went smoothly. However, I am unable to import the package and getting the error mentioned below. Could you please help? Thank you.!
My torch version is 1.12.0+cu113
import torch import deepwave
CalledProcessError Traceback (most recent call last) File ~/.conda/envs/openfwi/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1808, in _run_ninja_build(build_directory, verbose, error_prefix) 1807 stdout_fileno = 1 -> 1808 subprocess.run( 1809 command, 1810 stdout=stdout_fileno if verbose else subprocess.PIPE, 1811 stderr=subprocess.STDOUT, 1812 cwd=build_directory, 1813 check=True, 1814 env=env) 1815 except subprocess.CalledProcessError as e: 1816 # Python 2 and 3 compatible way of getting the error object.
File ~/.conda/envs/openfwi/lib/python3.8/subprocess.py:516, in run(input, capture_output, timeout, check, *popenargs, **kwargs) 515 if check and retcode: --> 516 raise CalledProcessError(retcode, process.args, 517 output=stdout, stderr=stderr) 518 return CompletedProcess(process.args, retcode, stdout, stderr)
CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last) Cell In[2], line 1 ----> 1 import deepwave
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/init.py:31 27 if torch.cuda.is_available(): 28 sources += [source_dir / 'scalar.cu', 29 source_dir / 'scalar_born.cu', 30 source_dir / 'elastic.cu'] ---> 31 torch.utils.cpp_extension.load( 32 name="deepwave", 33 sources=sources, 34 is_python_module=False, 35 extra_cflags=['-march=native', '-Ofast', '-fopenmp'], 36 extra_cuda_cflags=['--restrict', '-O3', '--use_fast_math'], 37 extra_ldflags=['-march=native', '-Ofast', '-fopenmp'] 38 )
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1202, in load(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates) 1111 def load(name, 1112 sources: Union[str, List[str]], 1113 extra_cflags=None, (...) 1121 is_standalone=False, 1122 keep_intermediates=True): 1123 r''' 1124 Loads a PyTorch C++ extension just-in-time (JIT). 1125 (...) 1200 verbose=True) 1201 ''' -> 1202 return _jit_compile( 1203 name, 1204 [sources] if isinstance(sources, str) else sources, 1205 extra_cflags, 1206 extra_cuda_cflags, 1207 extra_ldflags, 1208 extra_include_paths, 1209 build_directory or _get_build_directory(name, verbose), 1210 verbose, 1211 with_cuda, 1212 is_python_module, 1213 is_standalone, 1214 keep_intermediates=keep_intermediates)
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1425, in _jit_compile(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates) 1421 hipified_sources.add(hipify_result[s_abs]["hipified_path"] if s_abs in hipify_result else s_abs) 1423 sources = list(hipified_sources) -> 1425 _write_ninja_file_and_build_library( 1426 name=name, 1427 sources=sources, 1428 extra_cflags=extra_cflags or [], 1429 extra_cuda_cflags=extra_cuda_cflags or [], 1430 extra_ldflags=extra_ldflags or [], 1431 extra_include_paths=extra_include_paths or [], 1432 build_directory=build_directory, 1433 verbose=verbose, 1434 with_cuda=with_cuda, 1435 is_standalone=is_standalone) 1436 finally: 1437 baton.release()
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1537, in _write_ninja_file_and_build_library(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_standalone) 1535 if verbose: 1536 print(f'Building extension module {name}...') -> 1537 _run_ninja_build( 1538 build_directory, 1539 verbose, 1540 error_prefix=f"Error building extension '{name}'")
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1824, in _run_ninja_build(build_directory, verbose, error_prefix) 1822 if hasattr(error, 'output') and error.output: # type: ignore[union-attr] 1823 message += f": {error.output.decode(*SUBPROCESS_DECODE_ARGS)}" # type: ignore[union-attr] -> 1824 raise RuntimeError(message) from e
RuntimeError: Error building extension 'deepwave': [1/4] /cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/elastic.cu -o elastic.cuda.o FAILED: elastic.cuda.o /cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/elastic.cu -o elastic.cuda.o nvcc fatal : Could not open output file '/localscratch/1139367/tmpxft_00001ff5_0000000a' [2/4] /cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/scalar_born.cu -o scalar_born.cuda.o FAILED: scalar_born.cuda.o /cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/scalar_born.cu -o scalar_born.cuda.o nvcc fatal : Could not open output file '/localscratch/1139367/tmpxft_00001ff4_0000000a' [3/4] /cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/scalar.cu -o scalar.cuda.o FAILED: scalar.cuda.o /cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/scalar.cu -o scalar.cuda.o nvcc fatal : Could not open output file '/localscratch/1139367/tmpxft_00001ff3_0000000a' ninja: build stopped: subcommand failed.