How to get the output of a layer in the middle of a neural network?

[chang-change]

Dear Dr Lu Lu, For the two physical quantities A and B, the result of A will have a direct impact on B. So, I want to build a neural network like this, A is the output of a certain layer in the middle of the neural network, and B is the output of the last layer of the neural network. However, I don't know how can I get the value of A? Is there a better way to deal with this clearly causal variable in DeepXDE? Looking forward to your reply, thank you!

[praksharma]

May be you can obtain the weights here https://deepxde.readthedocs.io/en/latest/_modules/deepxde/model.html#LossHistory.set_loss_weights

Then simply do a forward pass manually. May be there is a better approach. This one should work.

[chang-change]

@praksharma Thanks for your response. I want to get the derivative of the intermediate variable, then construct the residual equation. Finally, adds the residual equation to the loss.

[praksharma]

Sorry, this is a vague description. Do you want to compute the residual from the output of the intermediate activation layer instead of the output layer? What do you want to achieve with this? Here is what I think.

• The intermediate layer should have exactly the same number of neurons/units as your output layer (please reply if you don't understand this). This means you are bottlenecking the gradient flow across the layers (read NTK-PINN or learning rate annealing PINN) to get some theoretical idea.
• Your PINN will never know any gradient information past that intermediate activation layer may result in random solution each time you train the model. This is what we call an ill-posed problem.
• DNNs are still blackbox curve fitting tool. Nobody can describe what will happen if you do this or that.

I don't think you can do this in DeepXDE. You need to create your own PINN and then modify it.

[chang-change]

@praksharma I'm so sorry I didn't describe it clearly. What I want to do is similar to this article.Physics-guided Design and Learning of Neural Networks for Predicting Drag Force on Particle Suspensions in Moving Fluids. However, there are some different.

• First, I want to use DNN instead of CNN.
• Second, I want to get the PDE of the output of the intermediate layer in respect to the space-time coordinate. Referred to as Loss 1.
• Third, there is an equation relationship between the output of the intermediate layer and the output of the output layer. I want to put this residual equation in loss function. Referred to as Loss 2. Maybe, there is a PDE of the output of output layer in respect to space-time coordinate. Referred to as Loss 3.
• Finally, Loss = Loss1 + Loss2 +Loss3 + Loss_icbc.

[chang-change]

[tsarikahin]

I think this is not supported by DeepXDE. You have to create your own architecture.

[praksharma]

I don't think this structure is robust for 2D and 3D problems. Did they publish this paper? By the way, this architecture is simple and you can write a code in 1 hour.

[chang-change]

@tsarikahin @praksharma Thanks for your reply. I know I can create the network by myself. I just want to know if the Deepxde can do it. The Deepxde is a magic tool, and it can save my time. By Deepxde, I can define geometry, boundary conditions, and training data, etc. It's convenient. But, I don't know that how to combine Deepxde with my network. Can you give me some suggestions? Thank you very much.

[lululxvi]

@praksharma @tsarikahin Good comments.

Actually it is doable in DeepXDE, although not in a straightforward way. See FAQ Q: A standard network doesn’t work for me, and I want to implement a network with a special structure/property.

[chang-change]

@lululxvi Thanks for your reply. I use apply_feature_transform to transform the inputs of network to the feature I need. The code as follows.

def input_feature_transform(x):

     xx = x[:,0:1]
     yy = x[:,1:2]
     zeta  = xx*(xx**2+yy**2+1)/(xx**2+yy**2)
     eta =  yy*(xx**2+yy**2-1)/(xx**2+yy**2)
     inputs = tf.concat([zeta,eta],axis = 1)
     return inputs
net.apply_feature_transform(input_feature_transform)  

How can I get the derivates of output of the network with respect to the features, instead of the derivates of output of the network with respect to spatial-time coordinate. Can you give me some suggestions? Thanks!

[lululxvi]

It is not straightforward to do. Then why don't you directly use zeta and eta as the network input?

[chang-change]

@lululxvi Thanks for your reply. I want to use Conformal Mapping to transform the problem of flowing around a cylinder to the problem of flowing around the wing. Because I don't know how to define the wing geometry by DeepXDE. Can you give me some advice? Thank you.

[lululxvi]

A few ideas:

• Use a polygon to approximate the geometry;
• Still use x and y as the input, but converting the derivative wrt zeta to the derivative wrt x and y
• Modify the source code so that you can compute the derivative wrt zeta

[chang-change]

@lululxvi Thanks for your reply. How can I use polygon to approximate the irregular shape?

[lululxvi]

See https://github.com/lululxvi/deepxde/issues/634