Adding specific internal training points with known solution

[maratsmuk]

Hi! First of all, thank you very much for such a great work, the project is very nice, strong and easy to understand and to use. I would like to ask the following issue. Suppose to have a time-dependent "standard" pde $u_t (x,t) + f(u, ux, u{xx}, x,t) = 0$ with Dirichlet bc u(-1,t) = u(1,t) = 0 and some initial conditions u(x,0) = g(x). Suppose that I have some additional points with the known solutions: for example, I know the values u(x_i, t_j) for some x_i and t_j > 0 (e.g., I have three additional data points x_i = [-0.5,0, 0.5], t_j = 0.1, so I know the values u(-0.5, 0.1), u(0,0.1), u(0.5,0.1)). How can I add this knowledge to the training set? Adding anchors to ICBC in this case just adds the values as collocation points (for them the values of the solution are not used, just the equation is checked). Fixing the solution function as input for PDE is also not an appropriate thing because I know the values of the solution only for these fixed points and not to all others. Probably, fixing manually data.train_y could work, but I have checked the code of training and it seems that these values are not used anywhere.

Follow up: The values x_i of additional points can be the same as in initial condition for simplicity. E.g., Some modification of the initial condition could solve this issue: for instance, initial conditions could return not only the values u(x,t0) but also the values u(x,t1). Can this be made? Thank you very much in advance!

[praksharma]

Well, You need to create a new MSE loss function. In PINN literature, this is called reconstruction error and these types of PINNs are often referred to as Sparse-Regulated PINNs. Although, It hardly makes sense to add reconstruction error, even in Nvidia Modulus they didn't use sparse regulation or any kind of cross-validation as well. The idea is to overfit the model on training data because we not going to test the model on new coordinates. If you want test new points, you need add lots of collocation points. For example for Lid-Driven Cavity Nvidia used 50000 Collocation points.

I guess you can simply modify this file to meet your demands. Just create your own input dataset. https://github.com/lululxvi/deepxde/blob/0352f8637bdf80e8f0904eba78b019148d8e93ca/examples/function/func.py#L1

[maratsmuk]

Well, You need to create a new MSE loss function. In PINN literature, this is called reconstruction error and these types of PINNs are often referred to as Sparse-Regulated PINNs. Although, It hardly makes sense to add reconstruction error, even in Nvidia Modulus they didn't use sparse regulation or any kind of cross-validation as well. The idea is to overfit the model on training data because we not going to test the model on new coordinates. If you want test new points, you need add lots of collocation points. For example for Lid-Driven Cavity Nvidia used 50000 Collocation points.

I guess you can simply modify this file to meet your demands. Just create your own input dataset.

Hi! Thank you very much for a very detailed and fast reply. The function you provided seems to be a good choice for me, thank you.

[camolina2]

Hi @praksharma and thank you so much for your information.

I would like to do the same experiment of @maratsmuk (I really appreciate if you find a solution)

I would like to solve a pde time-space depending and, add some information about the solution of the equation in some especific points.

So, I will have 4 losses function: boundary conditions, initial condition, pde and, the observation of the solution of the system.

Some of you have any idea how I can do this?

thanks in advance

Constanza Molina

[praksharma]

Hi @camolina2. It is straightforward. There is only one thing you must know: dde.data.PDE() object generates the datasets. If you go through the description of the class PDE() you will see the following.

Attributes:
train_x_all: A Numpy array of all points for training. `train_x_all` is
            unordered, and does not have duplication.
train_x: A Numpy array of the points fed into the network for training.
            `train_x` is constructed from `train_x_all`, ordered from BCs to PDE, and
            may have duplicate points.
train_x_bc: A Numpy array of the training points for BCs. `train_x_bc` is
            constructed from `train_x_all` at the first step of training, by default it
            won't be updated when `train_x_all` changes. To update `train_x_bc`, set it
            to `None` and call `bc_points`, and then update the loss function by
            ``model.compile()``.
num_bcs (list): `num_bcs[i]` is the number of points for `bcs[i]`.
test_x: A Numpy array of the points fed into the network for testing, ordered
            from BCs to PDE. The BC points are exactly the same points in `train_x_bc`.
train_aux_vars: Auxiliary variables that associate with `train_x`.
test_aux_vars: Auxiliary variables that associate with `test_x`.

You can vertically stack your input observation data for the sparse reconstruction after data = dde.data.PDE.

data.train_x = np.vstack([data.train_x, reconstruction_inputs])
data.train_x = data.train_x.astype("float32")

Just make sure the dimensions allow vertical stacking.

I literally spent hours finding where does @lululxvi generates the output dataset for BCs so that someone can simply stack the output observation data. Unfortunately, I could not find it in the data object or in the model object.

If someone can help in stacking the output observation data, it would be really helpful. Especially, there is no need to modify the source code.

The data object does have train_y as None type, even after training the model.

[maratsmuk]

Hi, @praksharma, thank you very much for your replies!

You can vertically stack your input observation data for the sparse reconstruction after data = dde.data.PDE.

It seems that @camolina2 mentioned a little bit different thing. Vertically stacking of train_x will only add new collocation points, while @camolina2 asked for new points with known solutions (to be honest, the same issue as I did: time-dependent PDE with some points with known solutions added to bc, ic and collocation points). A possible solution can be: stacking the points train_x_bc and changing num_bcs and using a more sophisticated bc function (e.g., for different values of t it will point to different functions). But I am not sure if it is possible to change manually train_x_bc (it is written that it is changed and generated from train_x_all during the training).

[lululxvi]

If you have the solution in some points, you can use PointSetBC, just as what we do for the inverse problem. Hope this what you need.

[maratsmuk]

@lululxvi , yes, thank you very much! Now it seems really what I needed. I would like to thank you very much again for such a great software, it really helps in my research.

[camolina2]

@lululxvi That works perfect! thank you so much!

Thank you both @praksharma, @maratsmuk for you help!!!

[praksharma]

@camolina2 Thanks for asking the question and saving my time. @lululxvi PointSetBC works as expected. Thanks for such a useful library. At least, you don't use the signed distance function as if we already know the solution as stated by Nvidia Modulus in their documentation.

Still, I have little confusion in DeepXDE. From my earlier post. if data = dde.data.PDE() then data.train_x denotes the input dataset. And I can check the input dataset easily. However data.train_y is None type.

Just wanted to know, are you saving the ground truth (BC solution + zeros vector for residual) somewhere? I am surely missing something, I looked at model.train() function definition but could not find any reference to data.train_y.

Can anyone point out the line in the source code where the output dataset is constructed?

[engsbk]

Is there an example on how to use PointSetBC()?

[praksharma]

https://deepxde.readthedocs.io/en/latest/search.html?q=+PointSetBC%28%29&check_keywords=yes&area=default

[engsbk]

Thank you! @praksharma, This is very informative, I hope it helps improving my accuracy.

[lululxvi]

Still, I have little confusion in DeepXDE. From my earlier post. if data = dde.data.PDE() then data.train_x denotes the input dataset. And I can check the input dataset easily. However data.train_y is None type.

Just wanted to know, are you saving the ground truth (BC solution + zeros vector for residual) somewhere? I am surely missing something, I looked at model.train() function definition but could not find any reference to data.train_y.

Can anyone point out the line in the source code where the output dataset is constructed?

data.train_y is the reference solution of the PDE (passed as data.PDE(..., solution=...)), but in the case where you don't provide the reference solution, then it is None. If we have the solution, then we can use data.train_y to compute a metric during network training, such as L2 relative error. Other information, such as BC solution and zero vector for residual, doesn't use data.train_y.

[engsbk]

I tried to use the same approach in https://deepxde.readthedocs.io/en/latest/demos/pinn_inverse/diffusion.1d.inverse.html?highlight=%20PointSetBC() to create a source point in a 2D-space 1D-time domain for the wave equation. my code for PointSet() looks like this:

Amp=2
freq=1
def Source_func(X_Sp):
    x = X_Sp[:,0:1] 
    y = X_Sp[:,1:2]
    t = X_Sp[:,2:3]
    return Amp*tf.sin(2*np.pi*freq*t)

X_source = np.vstack( (np.full((100), 0.5), np.full((100), 2.5), np.linspace(0, 5, num=100) )).T
Source_BC = dde.PointSetBC(X_source, Source_func(X_source))

My NN looks like this:

def pde(x, u):
    c = 1
    #Prep derivative terms
    du_xx = dde.grad.hessian(u, x, i=0, j=0)
    du_yy = dde.grad.hessian(u, x, i=1, j=1) 
    du_tt = dde.grad.hessian(u, x, i=2, j=2)

    #Return the PDE
    return   du_tt - (c ** 2)*(du_xx + du_yy)

data = dde.data.TimePDE(
    geomtime, pde,
    [],
    num_domain=5000,
    num_boundary=5000,
    num_initial=5000,
    train_distribution="pseudo",
)

n = 10
activation = f"LAAF-{n} silu"  # "LAAF-10 silu"
resampler = dde.callbacks.PDEResidualResampler(period=100)
net = dde.maps.FNN([3] + [32] * 5 + [1], activation, "Glorot normal")
net.apply_output_transform(transform)
model = dde.Model(data, net)
model.compile("adam", lr=1e-6)
model.train(epochs=10000, callbacks=[resampler], display_every=1000)
dde.optimizers.set_LBFGS_options(ftol=1e-10, maxiter=1e5)
model.compile("L-BFGS-B")
losshistory, train_state = model.train( callbacks=[resampler], model_save_path = "Trained_Models/Source_Model_Wave2D_HC")

My source point should be at location (0.5,2.5) on the spatial domain, but it seems like it is being applied all over the space domain. Any idea on how to fix this?

[lululxvi]

See examples https://deepxde.readthedocs.io/en/latest/demos/pinn_inverse.html for PointSetBC

[Wulx2050]

See examples https://deepxde.readthedocs.io/en/latest/demos/pinn_inverse.html for PointSetBC

deepxde.icbc.boundary_conditions.PointSetBC 是如此有用。很多时候只给定解条件(初值条件和边界条件)，PINN是很难训练的，特别是解由简单图像、单调递增递减变的更加复杂、混沌、不断震荡的时候。但是通过设置一些边界和内部的观察点(增加训练数据)，可以极大地加快收敛速度和模型拟合效果。这些观察点不仅仅只是在边界上，也可以在内部，所以这个方法的名字似乎很有歧义。

我这里有一个例子，展示了对 1d谐振子问题 只给初始条件数据、只给一个内部观察点数据、增加多个观察点数据和给一些错误的训练数据对PINN训练的影响。

https://github.com/Wulx2050/DeepXDE-and-PINN/blob/main/99%E7%89%A9%E7%90%86%E4%BF%A1%E6%81%AF%E7%A5%9E%E7%BB%8F%E7%BD%91%E7%BB%9C%E7%AE%80%E4%BB%8B.ipynb

那些gif图片(目前)被保存在这个目录下：https://github.com/Wulx2050/DeepXDE-and-PINN/tree/main/assets

我想您可以使用 DeepXDE 重写上面的1d谐振子问题，作为一个例子。

[lululxvi]

@Wulx2050 It would be great if you would like to add this example into DeepXDE examples.

[Wulx2050]

@Wulx2050 It would be great if you would like to add this example into DeepXDE examples.

我把代码发到 issues/645 上了，但是拟合结果似乎不太好，我找不到错误，麻烦您帮我检查一下代码。 I posted the code on issues/645, but the fitting result doesn't seem to be very good, I can't find the error, please check the code for me.