search

andruekonst / ConstraiNet

A PyTorch implementation of ConstraiNet – differentiable output constrained neural network layer
Apache License 2.0
4 stars 4 forks source link

Some questions about combining ConstraiNetLayer with regular torch-based neural networks #2

Open knochen336 opened 6 months ago

knochen336 commented 6 months ago

The ConstraiNetLayer you proposed is very good. I tried to add it to the neural network, but I found that this layer seems to be calculated only through CPU. Doesn't ConstraiNetLayer support CUDA and GPU? Another question is that numpy is used to define constraints in the ConstraiNetLayer layer. Will this cause the data format conversion between numpy and tensor to consume a lot of time? Because I found the following warning in my calculation example: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. Forgive me for not having deep mathematical knowledge :)

andruekonst commented 6 months ago

The underlying implementations are CUDA-compatible (it is based on pytorch tensors), for example: CenterDiameterBasedNN, ProjectionCenterDiameterNN. But we have not performed tests on GPU, as I remember.

When using ConstraiNetLayer, you can move the layer to GPU as usual after construction:

c_layer = ConstraiNetLayer(...).to('cuda')

The ConstraiNetLayer is just a wrapper, that simplifies interaction with the implementations. There are two steps: preparation and application (forward pass). At the first step some preparations are applied (find a feasible point, prepare null-space for equality constraints). Some of them operate only with NumPy, and there is no CUDA implementation available, so the whole preparation is done on CPU with NumPy arrays. The second step, which is a bottleneck and will be executed many times (each forward pass), is CUDA-compatible, as I've mentioned above.

About UserWarning: I'll try to locate this bug, there is definitely no reason to pass lists of NumPy arrays to torch.tensor(...). However, this should not affect the application step (forward pass).

Thank you for the question, it would help us a lot if you will write about your experience using CUDA (on success or failure)!

knochen336 commented 6 months ago

The underlying implementations are CUDA-compatible (it is based on pytorch tensors), for example: CenterDiameterBasedNN, ProjectionCenterDiameterNN. But we have not performed tests on GPU, as I remember.

When using ConstraiNetLayer, you can move the layer to GPU as usual after construction:

c_layer = ConstraiNetLayer(...).to('cuda')

The ConstraiNetLayer is just a wrapper, that simplifies interaction with the implementations. There are two steps: preparation and application (forward pass). At the first step some preparations are applied (find a feasible point, prepare null-space for equality constraints). Some of them operate only with NumPy, and there is no CUDA implementation available, so the whole preparation is done on CPU with NumPy arrays. The second step, which is a bottleneck and will be executed many times (each forward pass), is CUDA-compatible, as I've mentioned above.

About UserWarning: I'll try to locate this bug, there is definitely no reason to pass lists of NumPy arrays to torch.tensor(...). However, this should not affect the application step (forward pass).

Thank you for the question, it would help us a lot if you will write about your experience using CUDA (on success or failure)!

Thank you for your reply. I would be more than happy to give some feedback on my experience. I am using the ConstraiNetLayer layer with only the linear inequality constraint part. My python version is 3.8, torch version is 1.11.0, CUDA version 11.3, numpy version 1.21.6. First of all, ConstraiNetLayer can be computed on CUDA. However, the parameters passed into ProjectionCenterDiameterNN (e.g. self.constraint_set.linear.A) need to be moved to the GPU in advance. Otherwise you will get an error when calculating rays = zs - ps. Computing ConstraiNetLayer on CUDA reduces the running time of the neural network by a small amount, in my case the total training time was reduced from 90 minutes to 85 minutes. Then there's the warning about UserWarning: Creating a tensor from a list of numpy.ndarrays. This warning appears when the __init_implementation method of the ConstraiNetLayer class passes Cls torch.tensor( self.constraint_set.interior_point, dtype=torch.double). This warning can be avoided by modifying this code to torch.tensor(np.array(self.constraint_set.interior_point), dtype=torch.double). It's worth noting that such an operation doesn't seem to give a speed boost to the operation of the neural network. Finally I'd like to give feedback on two bugs I've encountered during use. 1, Please check the get_max_length method in layers such as CenterDiameterBasedNN, ProjectionCenterDiameterNN, and so on. The mu = numerator / dots in this method has the potential for a divide by 0 error, which can cause the loss in training to become nan. this should be avoided by modifying the 0 value to a very small positive number. For example dots = torch.where(dots == 0, torch.full_like(dots, 1e-8), dots) 2, when mode is selected as 'ray_shift', there may be data format inconsistency in def get_max_length method of CenterDiameterBasedNN (tensor.float vs. tensor.double), I added norm_rays = norm_rays.double() to avoid the problem.