Jittor / jittor

Jittor is a high-performance deep learning framework based on JIT compiling and meta-operators.
https://cg.cs.tsinghua.edu.cn/jittor/
Apache License 2.0
3.08k stars 311 forks source link

在固定了所有随机种子的情况下,jittor在每次运行训练代码时,每次的loss结果仍然不同 #381

Open yuyuyu223 opened 2 years ago

yuyuyu223 commented 2 years ago

我使用了如下代码固定随机种子

random.seed(seed) np.random.seed(seed) jt.set_global_seed(seed)

然而两次训练的结果存在一些偏差

第一次

epoch:0, iter:0, node_lr:0.001, edge_lr:0.001, node_loss:5.022258758544922, edge_loss:7.855024814605713, reg_node:0.0, reg_edge:0.0 epoch:0, iter:10, node_lr:0.001, edge_lr:0.001, node_loss:3.1473608016967773, edge_loss:2.6853835582733154, reg_node:0.0, reg_edge:0.0 epoch:0, iter:20, node_lr:0.001, edge_lr:0.001, node_loss:3.730393648147583, edge_loss:0.9033846855163574, reg_node:0.0, reg_edge:0.0 epoch:0, iter:30, node_lr:0.001, edge_lr:0.001, node_loss:3.8525800704956055, edge_loss:0.9183299541473389, reg_node:0.0, reg_edge:0.0 epoch:0, iter:40, node_lr:0.001, edge_lr:0.001, node_loss:3.51409912109375, edge_loss:0.654126763343811, reg_node:0.0, reg_edge:0.0

第二次

epoch:0, iter:0, node_lr:0.001, edge_lr:0.001, node_loss:5.022258281707764, edge_loss:7.855025768280029, reg_node:0.0, reg_edge:0.0 epoch:0, iter:10, node_lr:0.001, edge_lr:0.001, node_loss:3.1473841667175293, edge_loss:2.6853809356689453, reg_node:0.0, reg_edge:0.0 epoch:0, iter:20, node_lr:0.001, edge_lr:0.001, node_loss:3.730905771255493, edge_loss:0.9033833742141724, reg_node:0.0, reg_edge:0.0 epoch:0, iter:30, node_lr:0.001, edge_lr:0.001, node_loss:3.852440118789673, edge_loss:0.9183312058448792, reg_node:0.0, reg_edge:0.0 epoch:0, iter:40, node_lr:0.001, edge_lr:0.001, node_loss:3.5132908821105957, edge_loss:0.6541303992271423, reg_node:0.0, reg_edge:0.0

请问是否仍有某些flag未设置导致上述情况

diamond0910 commented 2 years ago

看上去你的loss精度误差是在万分位的,这是可以接受的吧?另外你可以尝试jt.seed(seed)。