search

official-stockfish / nnue-pytorch

Stockfish NNUE (Chess evaluation) trainer in Pytorch
GNU General Public License v3.0
320 stars 99 forks source link

No .nnue files found #261

Closed HenryZNNUE closed 12 months ago

HenryZNNUE commented 1 year ago

Hello, I'm a novice at NNUE-Pytorch. I started an experiment on Win11 and received an error after 3 epoches:

Training Comand: python easy_train.py --experiment-name L1-3328 --training-dataset=d:/nnue-pytorch-master/data/leela96-dfrc99-v2-T60novdecT77decT78jantosepT79aprmayT80juntonovjan-v6dd-T80febtomay2023.min.binpack --early-fen-skipping 27 --max_epoch 960 --lr 4.375e-4 --gamma 0.995 --start-lambda 1.0 --end-lambda 0.7 --tui False --engine-base-branch=official-stockfish/Stockfish/master --engine-test-branch=official-stockfish/Stockfish/master --gpus 0 --num-workers=4

Console Output: No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training Training run 0 exited unexpectedly. Error: Unknown error occured.

Can someone help me solve the problem?

Sopel97 commented 1 year ago

can you provide the logs? They are somewhere in the experiments directory

HenryZNNUE commented 1 year ago

Log: Engines provided. Enabling network testing. Stockfish already setup in D:\nnue-pytorch-master\scripts\easy_train_data\experiments/experiment_L1-3328\stockfish_base. Stockfish already setup in D:\nnue-pytorch-master\scripts\easy_train_data\experiments/experiment_L1-3328\stockfish_test. nnue-pytorch already setup in D:\nnue-pytorch-master\scripts\easy_train_data\experiments/experiment_L1-3328\nnue-pytorch Initialization completed. Doing network training on gpus [0]. 1 runs in total. Running network testing with command: ['C:\Users\Henry Z\AppData\Local\Programs\Python\Python311\python.exe', 'run_games.py', 'D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training', '--concurrency=10', '--explore_factor=1.5', '--c_chess_exe=D:\nnue-pytorch-master\scripts\easy_train_data\c-chess-cli\c-chess-cli.exe', '--stockfish_base=D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\stockfish_base\src\stockfish.exe', '--stockfish_test=D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\stockfish_test\src\stockfish.exe', '--book_file_name=D:\nnue-pytorch-master\scripts\easy_train_data\books\UHO_4060_v2.epd', '--hash=8', '--games_per_round=200', '--features=HalfKAv2_hm^', '--nodes_per_move=25000'] Also known as: C:\Users\Henry Z\AppData\Local\Programs\Python\Python311\python.exe run_games.py D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training --concurrency=10 --explore_factor=1.5 --c_chess_exe=D:\nnue-pytorch-master\scripts\easy_train_data\c-chess-cli\c-chess-cli.exe --stockfish_base=D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\stockfish_base\src\stockfish.exe --stockfish_test=D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\stockfish_test\src\stockfish.exe --book_file_name=D:\nnue-pytorch-master\scripts\easy_train_data\books\UHO_4060_v2.epd --hash=8 --games_per_round=200 --features=HalfKAv2_hm^ --nodes_per_move=25000 Running in working directory: D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\nnue-pytorch Running training with command: ['C:\Users\Henry Z\AppData\Local\Programs\Python\Python311\python.exe', 'train.py', 'd:\nnue-pytorch-master\data\leela96-dfrc99-v2-T60novdecT77decT78jantosepT79aprmayT80juntonovjan-v6dd-T80febtomay2023.min.binpack', 'd:\nnue-pytorch-master\data\leela96-dfrc99-v2-T60novdecT77decT78jantosepT79aprmayT80juntonovjan-v6dd-T80febtomay2023.min.binpack', '--num-workers=16', '--threads=2', '--max_epoch=960', '--batch-size=16384', '--random-fen-skipping=3', '--early-fen-skipping=27', '--gpus=0,', '--features=HalfKAv2_hm^', '--lr=0.0004375', '--gamma=0.995', '--lambda=1.0', '--network-save-period=20', '--save-last-network=True', '--seed=1', '--epoch-size=100000000', '--validation-size=1000000', '--default_root_dir=D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training\run_0', '--smart-fen-skipping', '--start-lambda=1.0', '--end-lambda=0.7', '--resume_from_checkpoint=D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training\run_0\lightning_logs\version_0\checkpoints\last.ckpt'] Also known as: C:\Users\Henry Z\AppData\Local\Programs\Python\Python311\python.exe train.py d:\nnue-pytorch-master\data\leela96-dfrc99-v2-T60novdecT77decT78jantosepT79aprmayT80juntonovjan-v6dd-T80febtomay2023.min.binpack d:\nnue-pytorch-master\data\leela96-dfrc99-v2-T60novdecT77decT78jantosepT79aprmayT80juntonovjan-v6dd-T80febtomay2023.min.binpack --num-workers=16 --threads=2 --max_epoch=960 --batch-size=16384 --random-fen-skipping=3 --early-fen-skipping=27 --gpus=0, --features=HalfKAv2_hm^ --lr=0.0004375 --gamma=0.995 --lambda=1.0 --network-save-period=20 --save-last-network=True --seed=1 --epoch-size=100000000 --validation-size=1000000 --default_root_dir=D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training\run_0 --smart-fen-skipping --start-lambda=1.0 --end-lambda=0.7 --resume_from_checkpoint=D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training\run_0\lightning_logs\version_0\checkpoints\last.ckpt Running in working directory: D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\nnue-pytorch C:\Users\Henry Z\AppData\Local\Programs\Python\Python311\Lib\site-packages\torchaudio\backend\utils.py:74: UserWarning: No audio backend is available. warnings.warn("No audio backend is available.") Global seed set to 1 Feature set: HalfKAv2_hm^ Num real features: 22528 Num virtual features: 768 Num features: 23296 Training with d:\nnue-pytorch-master\data\leela96-dfrc99-v2-T60novdecT77decT78jantosepT79aprmayT80juntonovjan-v6dd-T80febtomay2023.min.binpack validating with d:\nnue-pytorch-master\data\leela96-dfrc99-v2-T60novdecT77decT78jantosepT79aprmayT80juntonovjan-v6dd-T80febtomay2023.min.binpack Seed 1 Using batch size 16384 Smart fen skipping: True WLD fen skipping: True Random fen skipping: 3 Skip early plies: 27 Param index: 0 limiting torch to 2 threads. Using log dir D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training\run_0 ModelCheckpoint(save_last=True, save_top_k=-1, monitor=None) will duplicate the last checkpoint saved. C:\Users\Henry Z\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\connectors\accelerator_connector.py:478: LightningDeprecationWarning: Setting Trainer(gpus='0,') is deprecated in v1.7 and will be removed in v2.0. Please use Trainer(accelerator='gpu', devices='0,') instead. rank_zero_deprecation( C:\Users\Henry Z\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\connectors\checkpoint_connector.py:55: LightningDeprecationWarning: Setting Trainer(resume_from_checkpoint=) is deprecated in v1.5 and will be removed in v2.0. Please pass Trainer.fit(ckpt_path=) directly instead. rank_zero_deprecation( GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs C:\Users\Henry Z\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\trainer.py:1906: LightningDeprecationWarning: trainer.resume_from_checkpoint is deprecated in v1.5 and will be removed in v2.0. Specify the fit checkpoint path with trainer.fit(ckpt_path=) instead. rank_zero_deprecation( You are using a CUDA device ('NVIDIA GeForce RTX 4060 Laptop GPU') that has Tensor Cores. To properly utilize them, you should set torch.set_float32_matmul_precision('medium' | 'high') which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision Restoring states from the checkpoint path at D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training\run_0\lightning_logs\version_0\checkpoints\last.ckpt C:\Users\Henry Z\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py:338: UserWarning: The dirpath has changed from 'D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training\run_0\lightning_logs\version_0\checkpoints' to 'D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training\run_0\lightning_logs\version_1\checkpoints', therefore best_model_score, kth_best_model_path, kth_value, last_model_path and best_k_models won't be reloaded. Only best_model_path will be reloaded. warnings.warn( LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

| Name | Type | Params

0 | input | DoubleFeatureTransformerSlice | 47.9 M 1 | layer_stacks | LayerStacks | 303 K

48.2 M Trainable params 0 Non-trainable params 48.2 M Total params 192.808 Total estimated model params size (MB) Restored all states from the checkpoint file at D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training\run_0\lightning_logs\version_0\checkpoints\last.ckpt Using c++ data loader Ranger optimizer loaded. Gradient Centralization usage = False set state called

Sanity Checking: 0it [00:00, ?it/s]C:\Users\Henry Z\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\connectors\data_connector.py:224: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argument(try 16 which is the number of cpus on this machine) in theDataLoader` init to improve performance. rank_zero_warn(

Sanity Checking: 0%| | 0/2 [00:00<?, ?it/s] Sanity Checking DataLoader 0: 0%| | 0/2 [00:00<?, ?it/s] Sanity Checking DataLoader 0: 50%|█████ | 1/2 [00:01<00:01, 1.19s/it] Sanity Checking DataLoader 0: 100%|██████████| 2/2 [00:01<00:00, 1.66it/s] C:\Users\Henry Z\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\connectors\data_connector.py:224: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argument(try 16 which is the number of cpus on this machine) in theDataLoader` init to improve performance. rank_zero_warn(

Training: 6104it [00:00, ?it/s] Training: 0%| | 0/6166 [00:00<?, ?it/s] Epoch 2: 0%| | 0/6166 [00:00<?, ?it/s] Epoch 2: 0%| | 1/6166 [00:00<08:44, 11.74it/s] Epoch 2: 2%|▏ | 100/6166 [00:11<11:38, 8.68it/s, loss=0.00426, v_num=1] Epoch 2: 3%|▎ | 200/6166 [00:22<11:24, 8.71it/s, loss=0.00425, v_num=1] Epoch 2: 5%|▍ | 300/6166 [00:34<11:14, 8.70it/s, loss=0.00431, v_num=1] Epoch 2: 6%|▋ | 400/6166 [00:46<11:04, 8.68it/s, loss=0.00421, v_num=1] Epoch 2: 8%|▊ | 500/6166 [00:57<10:53, 8.67it/s, loss=0.00411, v_num=1] Epoch 2: 10%|▉ | 600/6166 [01:09<10:42, 8.66it/s, loss=0.00425, v_num=1] Epoch 2: 11%|█▏ | 700/6166 [01:20<10:31, 8.65it/s, loss=0.00422, v_num=1] Epoch 2: 13%|█▎ | 800/6166 [01:32<10:20, 8.65it/s, loss=0.00421, v_num=1] Epoch 2: 15%|█▍ | 900/6166 [01:44<10:08, 8.65it/s, loss=0.00411, v_num=1] Epoch 2: 16%|█▌ | 1000/6166 [01:55<09:57, 8.65it/s, loss=0.00429, v_num=1] Epoch 2: 18%|█▊ | 1100/6166 [02:07<09:45, 8.65it/s, loss=0.00415, v_num=1] Epoch 2: 19%|█▉ | 1200/6166 [02:18<09:34, 8.65it/s, loss=0.00413, v_num=1] Epoch 2: 21%|██ | 1300/6166 [02:30<09:22, 8.65it/s, loss=0.00413, v_num=1] Epoch 2: 23%|██▎ | 1400/6166 [02:41<09:11, 8.65it/s, loss=0.00404, v_num=1] Epoch 2: 24%|██▍ | 1500/6166 [02:53<08:59, 8.65it/s, loss=0.00414, v_num=1] Epoch 2: 26%|██▌ | 1600/6166 [03:04<08:47, 8.65it/s, loss=0.00415, v_num=1] Epoch 2: 28%|██▊ | 1700/6166 [03:16<08:36, 8.65it/s, loss=0.00415, v_num=1] Epoch 2: 29%|██▉ | 1800/6166 [03:28<08:24, 8.65it/s, loss=0.00412, v_num=1] Epoch 2: 31%|███ | 1900/6166 [03:39<08:13, 8.65it/s, loss=0.00408, v_num=1] Epoch 2: 32%|███▏ | 2000/6166 [03:51<08:01, 8.65it/s, loss=0.00413, v_num=1] Epoch 2: 34%|███▍ | 2100/6166 [04:02<07:50, 8.65it/s, loss=0.00408, v_num=1] Epoch 2: 36%|███▌ | 2200/6166 [04:14<07:38, 8.65it/s, loss=0.00422, v_num=1] Epoch 2: 37%|███▋ | 2300/6166 [04:25<07:26, 8.65it/s, loss=0.00401, v_num=1] Epoch 2: 39%|███▉ | 2400/6166 [04:37<07:15, 8.65it/s, loss=0.00402, v_num=1] Epoch 2: 41%|████ | 2500/6166 [04:48<07:03, 8.66it/s, loss=0.00413, v_num=1] Epoch 2: 42%|████▏ | 2600/6166 [05:00<06:51, 8.66it/s, loss=0.00413, v_num=1] Epoch 2: 44%|████▍ | 2700/6166 [05:11<06:40, 8.66it/s, loss=0.00406, v_num=1] Epoch 2: 45%|████▌ | 2800/6166 [05:23<06:28, 8.66it/s, loss=0.00405, v_num=1] Epoch 2: 47%|████▋ | 2900/6166 [05:34<06:17, 8.66it/s, loss=0.00405, v_num=1] Epoch 2: 49%|████▊ | 3000/6166 [05:46<06:05, 8.66it/s, loss=0.00406, v_num=1] Epoch 2: 50%|█████ | 3100/6166 [05:57<05:53, 8.66it/s, loss=0.00405, v_num=1] Epoch 2: 52%|█████▏ | 3200/6166 [06:09<05:42, 8.67it/s, loss=0.00406, v_num=1] Epoch 2: 54%|█████▎ | 3300/6166 [06:20<05:30, 8.67it/s, loss=0.00415, v_num=1] Epoch 2: 55%|█████▌ | 3400/6166 [06:32<05:18, 8.67it/s, loss=0.00404, v_num=1] Epoch 2: 57%|█████▋ | 3500/6166 [06:43<05:07, 8.67it/s, loss=0.00405, v_num=1] Epoch 2: 58%|█████▊ | 3600/6166 [06:54<04:55, 8.68it/s, loss=0.00396, v_num=1] Epoch 2: 60%|██████ | 3700/6166 [07:06<04:44, 8.68it/s, loss=0.00405, v_num=1] Epoch 2: 62%|██████▏ | 3800/6166 [07:17<04:32, 8.68it/s, loss=0.00398, v_num=1] Epoch 2: 63%|██████▎ | 3900/6166 [07:29<04:21, 8.68it/s, loss=0.00402, v_num=1] Epoch 2: 65%|██████▍ | 4000/6166 [07:40<04:09, 8.68it/s, loss=0.00406, v_num=1] Epoch 2: 66%|██████▋ | 4100/6166 [07:52<03:57, 8.68it/s, loss=0.00396, v_num=1] Epoch 2: 68%|██████▊ | 4200/6166 [08:03<03:46, 8.69it/s, loss=0.00399, v_num=1] Epoch 2: 70%|██████▉ | 4300/6166 [08:14<03:34, 8.69it/s, loss=0.004, v_num=1] Epoch 2: 71%|███████▏ | 4400/6166 [08:26<03:23, 8.69it/s, loss=0.00402, v_num=1] Epoch 2: 73%|███████▎ | 4500/6166 [08:37<03:11, 8.69it/s, loss=0.00401, v_num=1] Epoch 2: 75%|███████▍ | 4600/6166 [08:49<03:00, 8.69it/s, loss=0.00396, v_num=1] Epoch 2: 76%|███████▌ | 4700/6166 [09:00<02:48, 8.70it/s, loss=0.00393, v_num=1] Epoch 2: 78%|███████▊ | 4800/6166 [09:11<02:37, 8.70it/s, loss=0.00401, v_num=1] Epoch 2: 79%|███████▉ | 4900/6166 [09:23<02:25, 8.70it/s, loss=0.00399, v_num=1] Epoch 2: 81%|████████ | 5000/6166 [09:34<02:14, 8.70it/s, loss=0.00391, v_num=1] Epoch 2: 83%|████████▎ | 5100/6166 [09:46<02:02, 8.70it/s, loss=0.00391, v_num=1] Epoch 2: 84%|████████▍ | 5200/6166 [09:57<01:51, 8.70it/s, loss=0.0039, v_num=1] Epoch 2: 86%|████████▌ | 5300/6166 [10:09<01:39, 8.70it/s, loss=0.00397, v_num=1] Epoch 2: 88%|████████▊ | 5400/6166 [10:20<01:28, 8.70it/s, loss=0.00397, v_num=1] Epoch 2: 89%|████████▉ | 5500/6166 [10:31<01:16, 8.71it/s, loss=0.00398, v_num=1] Epoch 2: 91%|█████████ | 5600/6166 [10:43<01:05, 8.71it/s, loss=0.00398, v_num=1] Epoch 2: 92%|█████████▏| 5700/6166 [10:54<00:53, 8.71it/s, loss=0.00397, v_num=1] Epoch 2: 94%|█████████▍| 5800/6166 [11:05<00:42, 8.71it/s, loss=0.00389, v_num=1] Epoch 2: 96%|█████████▌| 5900/6166 [11:17<00:30, 8.71it/s, loss=0.00382, v_num=1] Epoch 2: 97%|█████████▋| 6000/6166 [11:28<00:19, 8.71it/s, loss=0.00383, v_num=1] Epoch 2: 99%|█████████▉| 6100/6166 [11:40<00:07, 8.71it/s, loss=0.00402, v_num=1]

Validation: 0it [00:00, ?it/s]

Validation: 0%| | 0/62 [00:00<?, ?it/s]

Validation DataLoader 0: 0%| | 0/62 [00:00<?, ?it/s]

Validation DataLoader 0: 2%|▏ | 1/62 [00:00<00:00, 330.96it/s]

Validation DataLoader 0: 3%|▎ | 2/62 [00:00<00:00, 181.48it/s]

Validation DataLoader 0: 5%|▍ | 3/62 [00:00<00:00, 130.32it/s]

Validation DataLoader 0: 6%|▋ | 4/62 [00:00<00:00, 106.40it/s]

Validation DataLoader 0: 8%|▊ | 5/62 [00:00<00:00, 112.11it/s]

Validation DataLoader 0: 10%|▉ | 6/62 [00:00<00:00, 105.49it/s]

Validation DataLoader 0: 11%|█▏ | 7/62 [00:00<00:00, 102.04it/s]

Validation DataLoader 0: 13%|█▎ | 8/62 [00:00<00:00, 103.08it/s]

Validation DataLoader 0: 15%|█▍ | 9/62 [00:00<00:00, 99.53it/s] 

Validation DataLoader 0: 16%|█▌ | 10/62 [00:00<00:00, 98.14it/s]

Validation DataLoader 0: 18%|█▊ | 11/62 [00:00<00:00, 98.66it/s]

Validation DataLoader 0: 19%|█▉ | 12/62 [00:00<00:00, 89.52it/s]

Validation DataLoader 0: 21%|██ | 13/62 [00:00<00:00, 88.55it/s]

Validation DataLoader 0: 23%|██▎ | 14/62 [00:00<00:00, 87.95it/s]

Validation DataLoader 0: 24%|██▍ | 15/62 [00:00<00:00, 90.06it/s]

Validation DataLoader 0: 26%|██▌ | 16/62 [00:00<00:00, 88.49it/s]

Validation DataLoader 0: 27%|██▋ | 17/62 [00:00<00:00, 87.93it/s]

Validation DataLoader 0: 29%|██▉ | 18/62 [00:00<00:00, 88.95it/s]

Validation DataLoader 0: 31%|███ | 19/62 [00:00<00:00, 81.48it/s]

Validation DataLoader 0: 32%|███▏ | 20/62 [00:00<00:00, 76.08it/s]

Validation DataLoader 0: 34%|███▍ | 21/62 [00:00<00:00, 71.23it/s]

Validation DataLoader 0: 35%|███▌ | 22/62 [00:00<00:00, 67.52it/s]

Validation DataLoader 0: 37%|███▋ | 23/62 [00:00<00:00, 64.49it/s]

Validation DataLoader 0: 39%|███▊ | 24/62 [00:00<00:00, 61.97it/s]

Validation DataLoader 0: 40%|████ | 25/62 [00:00<00:00, 59.82it/s]

Validation DataLoader 0: 42%|████▏ | 26/62 [00:00<00:00, 58.07it/s]

Validation DataLoader 0: 44%|████▎ | 27/62 [00:00<00:00, 56.49it/s]

Validation DataLoader 0: 45%|████▌ | 28/62 [00:00<00:00, 55.00it/s]

Validation DataLoader 0: 47%|████▋ | 29/62 [00:00<00:00, 53.75it/s]

Validation DataLoader 0: 48%|████▊ | 30/62 [00:00<00:00, 52.66it/s]

Validation DataLoader 0: 50%|█████ | 31/62 [00:00<00:00, 51.65it/s]

Validation DataLoader 0: 52%|█████▏ | 32/62 [00:00<00:00, 50.61it/s]

Validation DataLoader 0: 53%|█████▎ | 33/62 [00:00<00:00, 49.78it/s]

Validation DataLoader 0: 55%|█████▍ | 34/62 [00:00<00:00, 49.03it/s]

Validation DataLoader 0: 56%|█████▋ | 35/62 [00:00<00:00, 48.38it/s]

Validation DataLoader 0: 58%|█████▊ | 36/62 [00:00<00:00, 47.78it/s]

Validation DataLoader 0: 60%|█████▉ | 37/62 [00:00<00:00, 46.33it/s]

Validation DataLoader 0: 61%|██████▏ | 38/62 [00:00<00:00, 45.77it/s]

Validation DataLoader 0: 63%|██████▎ | 39/62 [00:00<00:00, 45.29it/s]

Validation DataLoader 0: 65%|██████▍ | 40/62 [00:00<00:00, 44.78it/s]

Validation DataLoader 0: 66%|██████▌ | 41/62 [00:00<00:00, 44.37it/s]

Validation DataLoader 0: 68%|██████▊ | 42/62 [00:00<00:00, 43.96it/s]

Validation DataLoader 0: 69%|██████▉ | 43/62 [00:00<00:00, 43.60it/s]

Validation DataLoader 0: 71%|███████ | 44/62 [00:01<00:00, 43.25it/s]

Validation DataLoader 0: 73%|███████▎ | 45/62 [00:01<00:00, 42.96it/s]

Validation DataLoader 0: 74%|███████▍ | 46/62 [00:01<00:00, 42.69it/s]

Validation DataLoader 0: 76%|███████▌ | 47/62 [00:01<00:00, 42.38it/s]

Validation DataLoader 0: 77%|███████▋ | 48/62 [00:01<00:00, 42.16it/s]

Validation DataLoader 0: 79%|███████▉ | 49/62 [00:01<00:00, 41.86it/s]

Validation DataLoader 0: 81%|████████ | 50/62 [00:01<00:00, 41.59it/s]

Validation DataLoader 0: 82%|████████▏ | 51/62 [00:01<00:00, 41.42it/s]

Validation DataLoader 0: 84%|████████▍ | 52/62 [00:01<00:00, 41.16it/s]

Validation DataLoader 0: 85%|████████▌ | 53/62 [00:01<00:00, 41.00it/s]

Validation DataLoader 0: 87%|████████▋ | 54/62 [00:01<00:00, 40.79it/s]

Validation DataLoader 0: 89%|████████▊ | 55/62 [00:01<00:00, 40.61it/s]

Validation DataLoader 0: 90%|█████████ | 56/62 [00:01<00:00, 40.45it/s]

Validation DataLoader 0: 92%|█████████▏| 57/62 [00:01<00:00, 40.21it/s]

Validation DataLoader 0: 94%|█████████▎| 58/62 [00:01<00:00, 40.10it/s]

Validation DataLoader 0: 95%|█████████▌| 59/62 [00:01<00:00, 39.92it/s]

Validation DataLoader 0: 97%|█████████▋| 60/62 [00:01<00:00, 39.75it/s]

Validation DataLoader 0: 98%|█████████▊| 61/62 [00:01<00:00, 39.63it/s]

Validation DataLoader 0: 100%|██████████| 62/62 [00:01<00:00, 39.48it/s]

 Epoch 2: 0%| | 0/6166 [00:00<?, ?it/s, loss=0.00402, v_num=1] Epoch 3: 0%| | 0/6166 [00:00<?, ?it/s, loss=0.00402, v_num=1] Epoch 3: 2%|▏ | 100/6166 [00:11<11:27, 8.82it/s, loss=0.0039, v_num=1] Epoch 3: 3%|▎ | 200/6166 [00:22<11:14, 8.84it/s, loss=0.00384, v_num=1] Epoch 3: 5%|▍ | 300/6166 [00:33<11:03, 8.83it/s, loss=0.00389, v_num=1] Epoch 3: 6%|▋ | 400/6166 [00:45<10:53, 8.83it/s, loss=0.004, v_num=1] Epoch 3: 8%|▊ | 500/6166 [00:56<10:42, 8.82it/s, loss=0.00391, v_num=1] Epoch 3: 10%|▉ | 600/6166 [01:08<10:31, 8.81it/s, loss=0.00398, v_num=1] Epoch 3: 11%|█▏ | 700/6166 [01:19<10:20, 8.81it/s, loss=0.0039, v_num=1] Epoch 3: 13%|█▎ | 800/6166 [01:30<10:08, 8.82it/s, loss=0.0039, v_num=1] Epoch 3: 15%|█▍ | 900/6166 [01:42<09:57, 8.82it/s, loss=0.00395, v_num=1] Epoch 3: 16%|█▌ | 1000/6166 [01:53<09:46, 8.81it/s, loss=0.00378, v_num=1] Epoch 3: 18%|█▊ | 1100/6166 [02:04<09:34, 8.81it/s, loss=0.00379, v_num=1] Epoch 3: 19%|█▉ | 1200/6166 [02:16<09:23, 8.81it/s, loss=0.00385, v_num=1] Epoch 3: 21%|██ | 1300/6166 [02:27<09:12, 8.81it/s, loss=0.00381, v_num=1] Epoch 3: 23%|██▎ | 1400/6166 [02:38<09:01, 8.81it/s, loss=0.00392, v_num=1] Epoch 3: 24%|██▍ | 1500/6166 [02:50<08:49, 8.81it/s, loss=0.00389, v_num=1] Epoch 3: 26%|██▌ | 1600/6166 [03:01<08:38, 8.80it/s, loss=0.0039, v_num=1] Epoch 3: 28%|██▊ | 1700/6166 [03:13<08:27, 8.81it/s, loss=0.00389, v_num=1] Epoch 3: 29%|██▉ | 1800/6166 [03:24<08:15, 8.80it/s, loss=0.00385, v_num=1] Epoch 3: 31%|███ | 1900/6166 [03:35<08:04, 8.80it/s, loss=0.00384, v_num=1] Epoch 3: 32%|███▏ | 2000/6166 [03:47<07:53, 8.81it/s, loss=0.00392, v_num=1] Epoch 3: 34%|███▍ | 2100/6166 [03:58<07:41, 8.81it/s, loss=0.00389, v_num=1] Epoch 3: 36%|███▌ | 2200/6166 [04:09<07:30, 8.81it/s, loss=0.00379, v_num=1] Epoch 3: 37%|███▋ | 2300/6166 [04:21<07:18, 8.81it/s, loss=0.00378, v_num=1] Epoch 3: 39%|███▉ | 2400/6166 [04:32<07:07, 8.81it/s, loss=0.0038, v_num=1] Epoch 3: 41%|████ | 2500/6166 [04:43<06:56, 8.81it/s, loss=0.00387, v_num=1] Epoch 3: 42%|████▏ | 2600/6166 [04:55<06:44, 8.81it/s, loss=0.00373, v_num=1] Epoch 3: 44%|████▍ | 2700/6166 [05:06<06:33, 8.81it/s, loss=0.00386, v_num=1] Epoch 3: 45%|████▌ | 2800/6166 [05:17<06:21, 8.81it/s, loss=0.00378, v_num=1] Epoch 3: 47%|████▋ | 2900/6166 [05:29<06:10, 8.81it/s, loss=0.00387, v_num=1] Epoch 3: 49%|████▊ | 3000/6166 [05:40<05:59, 8.82it/s, loss=0.00389, v_num=1] Epoch 3: 50%|█████ | 3100/6166 [05:51<05:47, 8.82it/s, loss=0.00372, v_num=1] Epoch 3: 52%|█████▏ | 3200/6166 [06:02<05:36, 8.82it/s, loss=0.00371, v_num=1] Epoch 3: 54%|█████▎ | 3300/6166 [06:14<05:25, 8.82it/s, loss=0.00379, v_num=1] Epoch 3: 55%|█████▌ | 3400/6166 [06:25<05:13, 8.82it/s, loss=0.00378, v_num=1] Epoch 3: 57%|█████▋ | 3500/6166 [06:36<05:02, 8.82it/s, loss=0.00375, v_num=1] Epoch 3: 58%|█████▊ | 3600/6166 [06:48<04:50, 8.82it/s, loss=0.00378, v_num=1] Epoch 3: 60%|██████ | 3700/6166 [06:59<04:39, 8.82it/s, loss=0.00379, v_num=1] Epoch 3: 62%|██████▏ | 3800/6166 [07:10<04:28, 8.82it/s, loss=0.00382, v_num=1] Epoch 3: 63%|██████▎ | 3900/6166 [07:21<04:16, 8.82it/s, loss=0.00393, v_num=1] Epoch 3: 65%|██████▍ | 4000/6166 [07:33<04:05, 8.82it/s, loss=0.00375, v_num=1] Epoch 3: 66%|██████▋ | 4100/6166 [07:44<03:54, 8.82it/s, loss=0.00378, v_num=1] Epoch 3: 68%|██████▊ | 4200/6166 [07:55<03:42, 8.83it/s, loss=0.00378, v_num=1] Epoch 3: 70%|██████▉ | 4300/6166 [08:07<03:31, 8.83it/s, loss=0.00379, v_num=1] Epoch 3: 71%|███████▏ | 4400/6166 [08:18<03:20, 8.83it/s, loss=0.00373, v_num=1] Epoch 3: 73%|███████▎ | 4500/6166 [08:29<03:08, 8.83it/s, loss=0.00371, v_num=1] Epoch 3: 75%|███████▍ | 4600/6166 [08:41<02:57, 8.83it/s, loss=0.0038, v_num=1] Epoch 3: 76%|███████▌ | 4700/6166 [08:52<02:46, 8.83it/s, loss=0.00365, v_num=1] Epoch 3: 78%|███████▊ | 4800/6166 [09:03<02:34, 8.83it/s, loss=0.00377, v_num=1] Epoch 3: 79%|███████▉ | 4900/6166 [09:15<02:23, 8.83it/s, loss=0.00375, v_num=1] Epoch 3: 81%|████████ | 5000/6166 [09:26<02:12, 8.83it/s, loss=0.00376, v_num=1] Epoch 3: 83%|████████▎ | 5100/6166 [09:37<02:00, 8.83it/s, loss=0.00366, v_num=1] Epoch 3: 84%|████████▍ | 5200/6166 [09:49<01:49, 8.83it/s, loss=0.00378, v_num=1] Epoch 3: 86%|████████▌ | 5300/6166 [10:00<01:38, 8.83it/s, loss=0.00377, v_num=1] Epoch 3: 88%|████████▊ | 5400/6166 [10:11<01:26, 8.83it/s, loss=0.0038, v_num=1] Epoch 3: 89%|████████▉ | 5500/6166 [10:22<01:15, 8.83it/s, loss=0.00376, v_num=1] Epoch 3: 91%|█████████ | 5600/6166 [10:34<01:04, 8.83it/s, loss=0.00365, v_num=1] Epoch 3: 92%|█████████▏| 5700/6166 [10:45<00:52, 8.83it/s, loss=0.00372, v_num=1] Epoch 3: 94%|█████████▍| 5800/6166 [10:56<00:41, 8.83it/s, loss=0.00363, v_num=1] Epoch 3: 96%|█████████▌| 5900/6166 [11:08<00:30, 8.83it/s, loss=0.00373, v_num=1] Epoch 3: 97%|█████████▋| 6000/6166 [11:19<00:18, 8.83it/s, loss=0.00368, v_num=1] Epoch 3: 99%|█████████▉| 6100/6166 [11:30<00:07, 8.83it/s, loss=0.00368, v_num=1]

Validation: 0it [00:00, ?it/s]

Validation: 0%| | 0/62 [00:00<?, ?it/s]

Validation DataLoader 0: 0%| | 0/62 [00:00<?, ?it/s]

Validation DataLoader 0: 2%|▏ | 1/62 [00:00<00:00, 333.01it/s]

Validation DataLoader 0: 3%|▎ | 2/62 [00:00<00:00, 153.80it/s]

Validation DataLoader 0: 5%|▍ | 3/62 [00:00<00:00, 136.36it/s]

Validation DataLoader 0: 6%|▋ | 4/62 [00:00<00:00, 133.32it/s]

Validation DataLoader 0: 8%|▊ | 5/62 [00:00<00:00, 138.88it/s]

Validation DataLoader 0: 10%|▉ | 6/62 [00:00<00:00, 130.42it/s]

Validation DataLoader 0: 11%|█▏ | 7/62 [00:00<00:00, 127.25it/s]

Validation DataLoader 0: 13%|█▎ | 8/62 [00:00<00:00, 116.74it/s]

Validation DataLoader 0: 15%|█▍ | 9/62 [00:00<00:00, 116.10it/s]

Validation DataLoader 0: 16%|█▌ | 10/62 [00:00<00:00, 114.26it/s]

Validation DataLoader 0: 18%|█▊ | 11/62 [00:00<00:00, 111.65it/s]

Validation DataLoader 0: 19%|█▉ | 12/62 [00:00<00:00, 107.61it/s]

Validation DataLoader 0: 21%|██ | 13/62 [00:00<00:00, 107.86it/s]

Validation DataLoader 0: 23%|██▎ | 14/62 [00:00<00:00, 108.09it/s]

Validation DataLoader 0: 24%|██▍ | 15/62 [00:00<00:00, 105.25it/s]

Validation DataLoader 0: 26%|██▌ | 16/62 [00:00<00:00, 102.69it/s]

Validation DataLoader 0: 27%|██▋ | 17/62 [00:00<00:00, 101.90it/s]

Validation DataLoader 0: 29%|██▉ | 18/62 [00:00<00:00, 92.39it/s] 

Validation DataLoader 0: 31%|███ | 19/62 [00:00<00:00, 84.13it/s]

Validation DataLoader 0: 32%|███▏ | 20/62 [00:00<00:00, 77.87it/s]

Validation DataLoader 0: 34%|███▍ | 21/62 [00:00<00:00, 73.18it/s]

Validation DataLoader 0: 35%|███▌ | 22/62 [00:00<00:00, 69.41it/s]

Validation DataLoader 0: 37%|███▋ | 23/62 [00:00<00:00, 66.10it/s]

Validation DataLoader 0: 39%|███▊ | 24/62 [00:00<00:00, 63.24it/s]

Validation DataLoader 0: 40%|████ | 25/62 [00:00<00:00, 60.90it/s]

Validation DataLoader 0: 42%|████▏ | 26/62 [00:00<00:00, 58.76it/s]

Validation DataLoader 0: 44%|████▎ | 27/62 [00:00<00:00, 57.08it/s]

Validation DataLoader 0: 45%|████▌ | 28/62 [00:00<00:00, 55.45it/s]

Validation DataLoader 0: 47%|████▋ | 29/62 [00:00<00:00, 54.10it/s]

Validation DataLoader 0: 48%|████▊ | 30/62 [00:00<00:00, 52.89it/s]

Validation DataLoader 0: 50%|█████ | 31/62 [00:00<00:00, 51.90it/s]

Validation DataLoader 0: 52%|█████▏ | 32/62 [00:00<00:00, 50.94it/s]

Validation DataLoader 0: 53%|█████▎ | 33/62 [00:00<00:00, 50.06it/s]

Validation DataLoader 0: 55%|█████▍ | 34/62 [00:00<00:00, 49.30it/s]

Validation DataLoader 0: 56%|█████▋ | 35/62 [00:00<00:00, 48.50it/s]

Validation DataLoader 0: 58%|█████▊ | 36/62 [00:00<00:00, 47.83it/s]

Validation DataLoader 0: 60%|█████▉ | 37/62 [00:00<00:00, 47.18it/s]

Validation DataLoader 0: 61%|██████▏ | 38/62 [00:00<00:00, 46.56it/s]

Validation DataLoader 0: 63%|██████▎ | 39/62 [00:00<00:00, 46.03it/s]

Validation DataLoader 0: 65%|██████▍ | 40/62 [00:00<00:00, 45.52it/s]

Validation DataLoader 0: 66%|██████▌ | 41/62 [00:00<00:00, 45.07it/s]

Validation DataLoader 0: 68%|██████▊ | 42/62 [00:00<00:00, 44.65it/s]

Validation DataLoader 0: 69%|██████▉ | 43/62 [00:00<00:00, 44.23it/s]

Validation DataLoader 0: 71%|███████ | 44/62 [00:01<00:00, 43.81it/s]

Validation DataLoader 0: 73%|███████▎ | 45/62 [00:01<00:00, 43.43it/s]

Validation DataLoader 0: 74%|███████▍ | 46/62 [00:01<00:00, 43.12it/s]

Validation DataLoader 0: 76%|███████▌ | 47/62 [00:01<00:00, 42.81it/s]

Validation DataLoader 0: 77%|███████▋ | 48/62 [00:01<00:00, 42.52it/s]

Validation DataLoader 0: 79%|███████▉ | 49/62 [00:01<00:00, 42.27it/s]

Validation DataLoader 0: 81%|████████ | 50/62 [00:01<00:00, 42.04it/s]

Validation DataLoader 0: 82%|████████▏ | 51/62 [00:01<00:00, 41.79it/s]

Validation DataLoader 0: 84%|████████▍ | 52/62 [00:01<00:00, 41.56it/s]

Validation DataLoader 0: 85%|████████▌ | 53/62 [00:01<00:00, 41.32it/s]

Validation DataLoader 0: 87%|████████▋ | 54/62 [00:01<00:00, 41.07it/s]

Validation DataLoader 0: 89%|████████▊ | 55/62 [00:01<00:00, 40.84it/s]

Validation DataLoader 0: 90%|█████████ | 56/62 [00:01<00:00, 40.63it/s]

Validation DataLoader 0: 92%|█████████▏| 57/62 [00:01<00:00, 40.47it/s]

Validation DataLoader 0: 94%|█████████▎| 58/62 [00:01<00:00, 40.26it/s]

Validation DataLoader 0: 95%|█████████▌| 59/62 [00:01<00:00, 40.11it/s]

Validation DataLoader 0: 97%|█████████▋| 60/62 [00:01<00:00, 39.95it/s]

Validation DataLoader 0: 98%|█████████▊| 61/62 [00:01<00:00, 39.80it/s]

Validation DataLoader 0: 100%|██████████| 62/62 [00:01<00:00, 39.66it/s]

 Epoch 3: 0%| | 0/6166 [00:00<?, ?it/s, loss=0.00368, v_num=1] Epoch 4: 0%| | 0/6166 [00:00<?, ?it/s, loss=0.00368, v_num=1] Epoch 4: 2%|▏ | 100/6166 [00:11<11:24, 8.86it/s, loss=0.00376, v_num=1] Epoch 4: 3%|▎ | 200/6166 [00:22<11:11, 8.88it/s, loss=0.00373, v_num=1] Epoch 4: 5%|▍ | 300/6166 [00:33<10:59, 8.89it/s, loss=0.00377, v_num=1] Epoch 4: 6%|▋ | 400/6166 [00:44<10:47, 8.91it/s, loss=0.00367, v_num=1] No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training No .nnue files found in D:\nnue-pytorch-master\scripts\easy_train_data\experiments\experiment_L1-3328\training Epoch 4: 8%|▊ | 500/6166 [00:56<10:35, 8.91it/s, loss=0.00371, v_num=1] Epoch 4: 10%|▉ | 600/6166 [01:07<10:24, 8.91it/s, loss=0.00376, v_num=1] Epoch 4: 11%|█▏ | 700/6166 [01:18<10:13, 8.91it/s, loss=0.00369, v_num=1] Epoch 4: 13%|█▎ | 800/6166 [01:29<10:02, 8.91it/s, loss=0.00374, v_num=1] Epoch 4: 15%|█▍ | 900/6166 [01:41<09:51, 8.90it/s, loss=0.00368, v_num=1] Epoch 4: 16%|█▌ | 1000/6166 [01:52<09:40, 8.90it/s, loss=0.0038, v_num=1] Epoch 4: 18%|█▊ | 1100/6166 [02:03<09:29, 8.90it/s, loss=0.00357, v_num=1] Epoch 4: 19%|█▉ | 1200/6166 [02:14<09:18, 8.90it/s, loss=0.00378, v_num=1] Epoch 4: 21%|██ | 1300/6166 [02:26<09:06, 8.90it/s, loss=0.00365, v_num=1] Epoch 4: 23%|██▎ | 1400/6166 [02:37<08:55, 8.89it/s, loss=0.00372, v_num=1] Training run 0 exited unexpectedly. Error: Unknown error occured.

Sopel97 commented 1 year ago

Hard to say, but it's unrelated to the "No .nnue files found". Something happened with torch/cuda.

HenryZNNUE commented 1 year ago

I have installed cuda 12.2, torch2.1.0dev+cu121 and pytorch-lightning 1.9.5 recently, and the problem still occured.