CUDA Runtime Error: an illegal memory access was encountered - Githubissues

BachiLi / diffvg

Differentiable Vector Graphics Rasterization

https://people.csail.mit.edu/tzumao/diffvg/

Apache License 2.0

923 stars 155 forks source link

CUDA Runtime Error: an illegal memory access was encountered #3

Open oeway opened 3 years ago

oeway commented 3 years ago

Hi, I encountered an error when running apps/single_curve_sdf_trans.py with GPU.

~/diffvg/apps$ CUDA_VISIBLE_DEVICES=1 python single_curve_sdf_trans.py 
CUDA Runtime Error: an illegal memory access was encountered at ~/diffvg/diffvg.cpp:1631

When this error occurred, the terminal hangs.

    // Clean up weight image
    if (scene->use_gpu) {
#ifdef __CUDACC__
        checkCuda(cudaFree(weight_image)); <--------------error occurred to this line
#else
        assert(false);
#endif
    } else {
        free(weight_image);
    }

BachiLi commented 3 years ago

I can't reproduce this. Why do you want to run this particular file? It's just for some random experiments with SDF rendering.

oeway commented 3 years ago

It was just an issue I came across when I was trying this particular file.

preddy5 commented 3 years ago

@BachiLi I am encountering the same issue as well, I think it might because of the number of shapes. In my scripts, the same code works as expected for 256 shapes but breaks if I am estimating 289 shapes.

BachiLi commented 3 years ago

is it a GPU specific issue? or does it also happen on CPU? i don't think i make this assumption anywhere in the code

preddy5 commented 3 years ago

I have changed all my input variables to the render function to CPU, it still raises the same error if I try to render more than 256 shape.

BachiLi commented 3 years ago

any chance you can build diffvg in debug mode and let me know at which line it breaks?

preddy5 commented 3 years ago

I dont know if this helps though, same error on cpu too.


     36 
     37       optim.zero_grad()
---> 38       loss.backward()
     39       optim.step()
     40 

~/anaconda3/envs/autosets/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    116                 products. Defaults to ``False``.
    117         """
--> 118         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    119 
    120     def register_hook(self, hook):

~/anaconda3/envs/autosets/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---> 93         allow_unreachable=True)  # allow_unreachable flag
     94 
     95 

RuntimeError: cuda runtime error (9) : invalid configuration argument at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/generic/THCTensorMath.cu:35

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/generic/THCTensorMath.cu line=35 error=9 : invalid configuration argument

preddy5 commented 3 years ago

Also the script raises the same error even with fewer shapes in debug mode.

BachiLi commented 3 years ago

Unfortuantely not helpful : ( Can you share which line in the C++ code it breaks in the CPU mode.

preddy5 commented 3 years ago

this is the line that the error is raised in normal mode, for more than 256 shapes CUDA Runtime Error: an illegal memory access was encountered at ~/diffvg/diffvg.cpp:1631 However in debug mode diffvg doesn't raise any error. The render returns an image as well, but backward pass raises an error for less than 256 shapes as well.

BachiLi commented 3 years ago

sorry i was in the wrong mindset: diffvg doesn't have CPU mode : ( can you run it with cuda-memcheck?

preddy5 commented 3 years ago

So far I have been working with jupyter notebook, when I run the code as a script I have new error, I think this might help,

/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [379,0,0], thread: [17,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [379,0,0], thread: [20,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [379,0,0], thread: [21,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [403,0,0], thread: [16,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [5,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [9,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [10,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [11,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [12,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [13,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [14,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [15,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [16,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [17,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [18,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [19,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [20,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [21,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [22,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [23,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [24,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [395,0,0], thread: [25,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [299,0,0], thread: [15,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [355,0,0], thread: [5,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [355,0,0], thread: [7,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [355,0,0], thread: [8,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [355,0,0], thread: [9,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [355,0,0], thread: [10,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [355,0,0], thread: [12,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [355,0,0], thread: [17,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [363,0,0], thread: [13,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [363,0,0], thread: [15,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [363,0,0], thread: [18,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [363,0,0], thread: [19,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [371,0,0], thread: [12,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [371,0,0], thread: [13,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [371,0,0], thread: [14,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [371,0,0], thread: [15,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [371,0,0], thread: [16,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [371,0,0], thread: [17,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [371,0,0], thread: [18,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [371,0,0], thread: [19,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [371,0,0], thread: [20,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [371,0,0], thread: [22,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [371,0,0], thread: [23,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [291,0,0], thread: [18,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [323,0,0], thread: [23,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [323,0,0], thread: [30,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [315,0,0], thread: [10,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [339,0,0], thread: [10,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [339,0,0], thread: [11,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [4,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [5,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [6,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [7,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [9,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [11,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [12,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [13,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [14,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [15,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [16,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [18,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [19,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [20,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [21,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [22,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [23,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [24,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [26,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [387,0,0], thread: [27,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [411,0,0], thread: [20,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [411,0,0], thread: [21,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [419,0,0], thread: [11,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [427,0,0], thread: [13,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [347,0,0], thread: [7,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [435,0,0], thread: [12,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [443,0,0], thread: [12,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [331,0,0], thread: [27,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [283,0,0], thread: [10,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [307,0,0], thread: [14,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [370,0,0], thread: [61,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [378,0,0], thread: [61,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [354,0,0], thread: [63,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [322,0,0], thread: [62,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [338,0,0], thread: [59,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [346,0,0], thread: [57,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [314,0,0], thread: [62,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [298,0,0], thread: [62,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [418,0,0], thread: [61,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [410,0,0], thread: [61,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [402,0,0], thread: [61,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [306,0,0], thread: [55,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [306,0,0], thread: [57,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [306,0,0], thread: [62,0,0] Assertion `num_fragments < max_hit_shapes` failed.
/home/creddy/Work/diffvg/diffvg.cpp:556: TVector4<float> sample_color(const SceneData &, const TVector4<float> *, const TVector2<float> &, const TVector4<float> *, EdgeQuery *, TVector4<float> *, float *): block: [330,0,0], thread: [59,0,0] Assertion `num_fragments < max_hit_shapes` failed.
CUDA Runtime Error: device-side assert triggered at /home/creddy/Work/diffvg/diffvg.cpp:1594

BachiLi commented 3 years ago

Change max_hit_shapes to a larger number? https://github.com/BachiLi/diffvg/blob/fd38f71527b6258ddaf54ba2db3596b396dc80e1/diffvg.cpp#L542

Sorry this is a stupid restriction. I need to fix it : (

preddy5 commented 3 years ago

@BachiLi works like a charm, thank you for being responsive.

MADONOKOUKI commented 1 year ago

Does anyone happen similar situation at backpropagation? (no error on CPU but error on GPU) Ubuntu 18.04 cuda 10.0 cudnn 7.0 It might be due to a large amount of parameters (Just I use one GPU since I can't parallel the GPUs now).

iteration: 0
Scene construction, time: 40.43806 s
Forward pass, time: 0.00457 s
cuda:0 torch.float32 torch.float32 cuda:0
Traceback (most recent call last):
  File "main.py", line 217, in <module>
    loss.backward()
  File "/root/anaconda3/envs/py37/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/root/anaconda3/envs/py37/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/root/anaconda3/envs/py37/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/myname/adobe/tmp/meshgrad/pydiffvg/render_pytorch.py", line 556, in backward
    eval_positions.shape[0])
RuntimeError
CUDA Runtime Error: an illegal memory access was encountered at /home/myname/adobe/tmp/meshgrad/scene.cpp:1523