Custom dataset config options

[Sazoji]

I've been able to run this on my own data (slowly, on a 3070) using U^2Net matting and colmap2nerf code from instant-ngp (if others are following this, remove the image extensions, and it should run on windows) The issue I'm having is val, and test data or the lack of it. Are there config options to remove the requirement for those datasets. Copying and renaming the json lists to val and train works, but is rather cumbersome, and I was wondering if I was missing a better option with the NeRD dataset preparation, which use manual masks not applied to the image itself (which is possible with U^2 net, and would help with colmap mapping since the background can still be used for feature tracking.)

I haven't looked into what data is required to be presented in the config, nor resolution/training options, but I'm just wondering what's the generally intended arguments and method for presenting your own data, when no val is available.

[jmunkberg]

Nice to see that you got it running on your data!

It seems that the bbox is a bit too small, as the upper part of the model is cut. You can either increase the mesh scale parameter in the .json config "mesh_scale" : 2.5, (use a value that tightly fits your model) or add a scaling to your transform matrix in the dataset. We start from a tetrahedral grid, and if that grid is smaller than the model to capture, the result can be cut as shown above

For the dataset question, I would just modify or create a custom dataloader, by modifying the class DatasetNERF https://github.com/NVlabs/nvdiffrec/blob/main/dataset/dataset_nerf.py . That gives you full freedom w.r.t. how to apply the masks, etc. There is currently no option exposed in the config to remove validation data. The flag parser.add_argument('--validate', type=bool, default=True) currently controls if we should run an inference pass on all the validation/test data after training.

The train and validation data for nerf are loaded here https://github.com/NVlabs/nvdiffrec/blob/main/train.py#L563 , so an easy WAR is to just load transforms_train.json also for the validation/test data (or like you did, copy the series).

[JHnvidia]

Unfortunately the code wasn't written with that in mind, so there's no configuration flag.

You can find the setup code for the nerf_synthetic dataset here: https://github.com/NVlabs/nvdiffrec/blob/main/train.py#L564. You should be able to just change 'transforms_test.json' to 'transforms_train.json' to use the same training and validation/test set, or add a check if the file exists

if os.path.exists(os.path.join(FLAGS.ref_mesh, 'transforms_test.json')):
    dataset_validate = DatasetNERF(os.path.join(FLAGS.ref_mesh, 'transforms_test.json'), FLAGS)
else:
    dataset_validate = DatasetNERF(os.path.join(FLAGS.ref_mesh, 'transforms_train.json'), FLAGS)

For NeRD datasets (a few lines up) we always use the same train & test set.

[Sazoji]

I'll look into how imgs2poses could be adapted to produce rembg/U^2net masks, make it a single script like colmap2nerf. Is the mask handled as binary or a full depth matte in nvdiffrec? Automatic trimapping/refinement might be a waste otherwise.

[Sazoji]

issue solved, know where to look to automatically produce custom datasets from colmap data and some config options to improve the mesh scale and load poses from a single file.

[eyebies]

@jmunkberg Just to wanted to cross check the steps for custom data

1. Generate masks (u2net or rembg)
2. generate poses img2poses
3. create soft link for transforms_test.json
4. Adjust "mesh_scale" @mjc619 Would it be possible to share the script :) ?

[jmunkberg]

@eyebies Yes, that is correct. The same procedure as when generating data for NeRF/NeRD applies. If you want to use synthetic data, you can follow the Blender setup used in the synthetic NeRF data: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1 in which case you get exact masks and perfect poses.

For masks, the methods you cite above would work. We tried with a bunch of other (Detectron2, Photoshop), see e.g., Fig 23 in the paper: https://nvlabs.github.io/nvdiffrec/assets/paper.pdf In general, high quality masks improve results, particularly the geometry reconstruction.

For poses, COLMAP works well in our tests. We currently do not optimize over poses or correct for noise in the pose estimation, but that could likely be added.

[Sazoji]

@eyebies img2poses would produce a .npy file since it's using a different dataset format (and the benefit of not having to make test and val files), but both are loading the same data from colmap.

Other than that, yea. I'll be sure to link it in this issue. Want it to be a single gist.

[Sazoji]

@eyebies

   @mjc619 Would it be possible to share the script :) ?

https://gist.github.com/mjc619/e20835d652c51f305ce328342af7fefd comment any issues, the script should be self-contained and not need any exotic libraries other than interfacing with colmap and optionally ffmpeg or rembg.

[eyebies]

thanks a lot guys ! I really appreciate the gesture ! Just for the sake of completeness. Here are a couple of algorithms that could be used in-place of the rembg. These are mostly class agnostic. https://github.com/Karel911/TRACER https://github.com/hkchengrex/CascadePSP

Thanks again !

[Sazoji]

Depending on how the masking is handled, those are probably far better. rembg included applying alpha mattes to the png result, which gave a simple solution for masking NeRF datasets. The U^2 net masks aren't binary seg images (nor SOTA), and IDK how nvdiffrec would handle the masks and if a hard or uncertain edge produces a better result. More to explore, others had ideas for using guided sequence segmentation for video, the refinement model would really help with that!

[c6s0]

@mjc619 Hey, i would also like to train with custom data. However, I don't know my way around that well and couldn't follow the thread here that well... I have also trained some scenes with instant ngp as you described. Could you tell me the steps how I could train my own data (images) with this repo to generate 3D models?

[Sazoji]

@c6s0 I'll probably fork, two characters in the LLFF dataloader make it a lot harder to use NeRD datasets than it needs to be, and a lot of the masking probably requires its own scripts to work well. Making a tutorial.md on the fork wouldn't be impossible once it's optimized for a 8gb graphics card.

Colmap2poses is still a work in progress, I can now work video directly out of my phone, but the results are lackluster. I also need to optimize some losses in the training and find an alternative to ffmpeg that finds clear frames to use in rendering handheld video (colmap seems to just accept a lot of the blurry frames in sequential mode).

I'd recommend just using nerf datasets made for instant-ngp with a manual alpha mask. Rendering video -> 3d model on a consumer gpu is possible, just not that friendly or high quality yet.

[c6s0]

@mjc619 Thank you for your answer! A little tutorial in the repo would be super cool and helpful! GPU power shouldn't be a problem (at least in my case) since I have access to 3090's and A100's. But yes, probably it just needs some more time.

[Sazoji]

Motion blur and keeping the model in the center of frame for mapping seem to still be the major reasons for models turning into swiss cheese 5k iters in. Unlike the DSLR results, which look pretty good after training, I can't change the exposure time with phone video. If you're trying to set up your own dataset right now, use camera photos and filter by quality and center. @c6s0, running colmap manually and using stabilized (and low exposure time) images is all you need, failures after colmaping are due to the "images" folder holding more images than what colmap accepted in the dataset, the latest version of colmap2poses should make a .npy file and a new text document of all the images the dataset is actually using, since the minified dataset file can't be read easily. My fork of nvdiffrec skips the "too many images within the folder" sanity check and should work on whatever NeRF sets you're converting to LLFF, as long as you have clean masks, sharper binary masks seem to work better than the alpha ones. New datasets made from video are still producing weird results late in training though.

Cv2 is already called for the NeRF datasets, so I'm thinking of ways to prefilter images before colmap based on sharpness, best recording settings to get sharp images (60fps, ffmpeg to resize down to 520p), and possibly using the mask to find out how close to center the object is. @jmunkberg I'm also getting a vram leak during training, is there a reason why vram should increase after ~2h of training?

[jmunkberg]

@mjc619 No, vram should not increase over time! That said, when we switch from volumetric texturing (using the MLP) to traditional 2D textures, memory consumption changes, depending on the texture resolution, MLP parameters etc. We also have the option to run with depth peeling on in the second pass (used in NeRF mic, ficus, drums, ship). This is also increase memory drastically. This happens halfway into the training. Could you share some details? Is this a slow increase over time or a sudden change?

[Sazoji]

log showing 2.6gb free turn to 0 in under an hour

```(dmodel) E:\test4video\nvdiffrec>python train.py --config configs/cup.json Config / Flags: --------- config configs/cup.json iter 10000 batch 1 spp 1 layers 1 train_res [1422, 800] display_res [1422, 800] texture_res [2048, 2048] display_interval 0 save_interval 100 learning_rate [0.02, 0.003] min_roughness 0.08 custom_mip False random_textures True background white loss logl1 out_dir out/salt ref_mesh data/salt base_mesh None validate False mtl_override None dmtet_grid 128 mesh_scale 2.8 env_scale 1.0 envmap None display [{'latlong': True}, {'bsdf': 'kd'}, {'bsdf': 'ks'}, {'bsdf': 'normal'}] camera_space_light False lock_light False lock_pos False sdf_regularizer 0.2 laplace relative laplace_scale 10000 pre_load True kd_min [0.0, 0.0, 0.0, 0.0] kd_max [1.0, 1.0, 1.0, 1.0] ks_min [0.0, 0.08, 0.0] ks_max [1.0, 1.0, 1.0] nrm_min [-1.0, -1.0, 0.0] nrm_max [1.0, 1.0, 1.0] cam_near_far [0.1, 1000.0] learn_light True local_rank 0 multi_gpu False --------- DatasetLLFF: 87 images with shape [1422, 800] DatasetLLFF: auto-centering at [0.27549016 1.324876 3.206055 ] DatasetLLFF: 87 images with shape [1422, 800] DatasetLLFF: auto-centering at [0.27549016 1.324876 3.206055 ] Encoder output: 32 dims Using C:\Users\Matt\AppData\Local\torch_extensions\torch_extensions\Cache\py39_cu113 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file C:\Users\Matt\AppData\Local\torch_extensions\torch_extensions\Cache\py39_cu113\renderutils_plugin\build.ninja... Building extension module renderutils_plugin... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module renderutils_plugin... iter= 0, img_loss=0.368703, reg_loss=0.333922, lr=0.01999, time=364.3 ms, rem=1.01 h free gpu memory: [2627] iter= 10, img_loss=0.231970, reg_loss=0.325075, lr=0.01990, time=329.7 ms, rem=54.90 m free gpu memory: [1361] iter= 20, img_loss=0.108235, reg_loss=0.313862, lr=0.01981, time=325.9 ms, rem=54.21 m free gpu memory: [1361] iter= 30, img_loss=0.067148, reg_loss=0.300575, lr=0.01972, time=327.2 ms, rem=54.37 m free gpu memory: [1361] iter= 40, img_loss=0.026152, reg_loss=0.289282, lr=0.01963, time=329.8 ms, rem=54.75 m free gpu memory: [1361] iter= 50, img_loss=0.078296, reg_loss=0.283187, lr=0.01954, time=328.0 ms, rem=54.39 m free gpu memory: [1361] iter= 60, img_loss=0.053913, reg_loss=0.280181, lr=0.01945, time=324.7 ms, rem=53.79 m free gpu memory: [1359] iter= 70, img_loss=0.076872, reg_loss=0.278783, lr=0.01936, time=325.6 ms, rem=53.89 m free gpu memory: [1357] iter= 80, img_loss=0.057660, reg_loss=0.277743, lr=0.01927, time=325.5 ms, rem=53.82 m free gpu memory: [1357] iter= 90, img_loss=0.055019, reg_loss=0.276530, lr=0.01918, time=324.1 ms, rem=53.53 m free gpu memory: [1357] iter= 100, img_loss=0.061880, reg_loss=0.275087, lr=0.01909, time=326.5 ms, rem=53.87 m free gpu memory: [1357] iter= 110, img_loss=0.064729, reg_loss=0.273679, lr=0.01900, time=325.6 ms, rem=53.67 m free gpu memory: [1357] iter= 120, img_loss=0.086429, reg_loss=0.272541, lr=0.01892, time=328.3 ms, rem=54.06 m free gpu memory: [1357] iter= 130, img_loss=0.053687, reg_loss=0.271435, lr=0.01883, time=331.1 ms, rem=54.47 m free gpu memory: [1356] iter= 140, img_loss=0.072052, reg_loss=0.270332, lr=0.01874, time=340.1 ms, rem=55.89 m free gpu memory: [1356] iter= 150, img_loss=0.060201, reg_loss=0.269349, lr=0.01866, time=356.6 ms, rem=58.55 m free gpu memory: [1355] iter= 160, img_loss=0.089421, reg_loss=0.268333, lr=0.01857, time=340.6 ms, rem=55.86 m free gpu memory: [1355] iter= 170, img_loss=0.090497, reg_loss=0.267119, lr=0.01849, time=339.1 ms, rem=55.56 m free gpu memory: [1355] iter= 180, img_loss=0.036824, reg_loss=0.265990, lr=0.01840, time=341.7 ms, rem=55.93 m free gpu memory: [1347] iter= 190, img_loss=0.042263, reg_loss=0.265294, lr=0.01832, time=333.8 ms, rem=54.58 m free gpu memory: [1347] iter= 200, img_loss=0.030886, reg_loss=0.264061, lr=0.01823, time=335.6 ms, rem=54.82 m free gpu memory: [1355] iter= 210, img_loss=0.067843, reg_loss=0.262727, lr=0.01815, time=347.0 ms, rem=56.62 m free gpu memory: [1357] iter= 220, img_loss=0.048636, reg_loss=0.261332, lr=0.01806, time=331.5 ms, rem=54.03 m free gpu memory: [1357] iter= 230, img_loss=0.069288, reg_loss=0.260106, lr=0.01798, time=333.6 ms, rem=54.32 m free gpu memory: [1357] iter= 240, img_loss=0.079395, reg_loss=0.258883, lr=0.01790, time=338.0 ms, rem=54.98 m free gpu memory: [1365] iter= 250, img_loss=0.071042, reg_loss=0.257616, lr=0.01782, time=332.3 ms, rem=54.00 m free gpu memory: [1365] iter= 260, img_loss=0.048266, reg_loss=0.256596, lr=0.01773, time=337.4 ms, rem=54.77 m free gpu memory: [1365] iter= 270, img_loss=0.057781, reg_loss=0.255818, lr=0.01765, time=337.1 ms, rem=54.67 m free gpu memory: [1364] iter= 280, img_loss=0.045252, reg_loss=0.254978, lr=0.01757, time=330.7 ms, rem=53.57 m free gpu memory: [1364] iter= 290, img_loss=0.058800, reg_loss=0.254186, lr=0.01749, time=329.5 ms, rem=53.32 m free gpu memory: [1364] iter= 300, img_loss=0.050951, reg_loss=0.253028, lr=0.01741, time=330.6 ms, rem=53.45 m free gpu memory: [1364] iter= 310, img_loss=0.082238, reg_loss=0.251990, lr=0.01733, time=350.9 ms, rem=56.67 m free gpu memory: [1355] iter= 320, img_loss=0.058590, reg_loss=0.251026, lr=0.01725, time=333.0 ms, rem=53.72 m free gpu memory: [1360] iter= 330, img_loss=0.049502, reg_loss=0.249718, lr=0.01717, time=337.0 ms, rem=54.31 m free gpu memory: [1368] iter= 340, img_loss=0.080593, reg_loss=0.248403, lr=0.01709, time=332.6 ms, rem=53.55 m free gpu memory: [1368] iter= 350, img_loss=0.048158, reg_loss=0.247322, lr=0.01701, time=331.7 ms, rem=53.35 m free gpu memory: [1368] iter= 360, img_loss=0.032747, reg_loss=0.246358, lr=0.01694, time=332.0 ms, rem=53.34 m free gpu memory: [1368] iter= 370, img_loss=0.058038, reg_loss=0.245217, lr=0.01686, time=329.6 ms, rem=52.90 m free gpu memory: [1368] iter= 380, img_loss=0.067462, reg_loss=0.244010, lr=0.01678, time=331.6 ms, rem=53.17 m free gpu memory: [1368] iter= 390, img_loss=0.070972, reg_loss=0.242969, lr=0.01670, time=330.5 ms, rem=52.94 m free gpu memory: [1368] iter= 400, img_loss=0.064048, reg_loss=0.241791, lr=0.01663, time=331.7 ms, rem=53.07 m free gpu memory: [1368] iter= 410, img_loss=0.060880, reg_loss=0.240637, lr=0.01655, time=341.0 ms, rem=54.50 m free gpu memory: [1368] iter= 420, img_loss=0.031116, reg_loss=0.239542, lr=0.01648, time=332.1 ms, rem=53.03 m free gpu memory: [1368] iter= 430, img_loss=0.060647, reg_loss=0.238729, lr=0.01640, time=340.3 ms, rem=54.28 m free gpu memory: [1360] iter= 440, img_loss=0.023712, reg_loss=0.237723, lr=0.01632, time=330.3 ms, rem=52.63 m free gpu memory: [1360] iter= 450, img_loss=0.033039, reg_loss=0.236682, lr=0.01625, time=333.2 ms, rem=53.03 m free gpu memory: [1360] iter= 460, img_loss=0.055331, reg_loss=0.235496, lr=0.01617, time=332.0 ms, rem=52.79 m free gpu memory: [1360] iter= 470, img_loss=0.045686, reg_loss=0.234248, lr=0.01610, time=334.5 ms, rem=53.13 m free gpu memory: [1360] iter= 480, img_loss=0.080620, reg_loss=0.233008, lr=0.01603, time=345.0 ms, rem=54.74 m free gpu memory: [1368] iter= 490, img_loss=0.095046, reg_loss=0.232004, lr=0.01595, time=344.6 ms, rem=54.62 m free gpu memory: [1368] iter= 500, img_loss=0.037334, reg_loss=0.230876, lr=0.01588, time=334.0 ms, rem=52.88 m free gpu memory: [1368] iter= 510, img_loss=0.027632, reg_loss=0.230020, lr=0.01581, time=341.0 ms, rem=53.94 m free gpu memory: [1220] iter= 520, img_loss=0.021546, reg_loss=0.229151, lr=0.01573, time=329.2 ms, rem=52.01 m free gpu memory: [1220] iter= 530, img_loss=0.044913, reg_loss=0.228139, lr=0.01566, time=333.3 ms, rem=52.61 m free gpu memory: [1220] iter= 540, img_loss=0.033730, reg_loss=0.226879, lr=0.01559, time=331.9 ms, rem=52.33 m free gpu memory: [1220] iter= 550, img_loss=0.058484, reg_loss=0.225574, lr=0.01552, time=333.7 ms, rem=52.56 m free gpu memory: [1220] iter= 560, img_loss=0.052862, reg_loss=0.224318, lr=0.01545, time=347.1 ms, rem=54.61 m free gpu memory: [1228] iter= 570, img_loss=0.049570, reg_loss=0.223177, lr=0.01538, time=344.7 ms, rem=54.18 m free gpu memory: [1228] iter= 580, img_loss=0.040358, reg_loss=0.222171, lr=0.01530, time=342.8 ms, rem=53.82 m free gpu memory: [1228] iter= 590, img_loss=0.032445, reg_loss=0.221147, lr=0.01523, time=332.3 ms, rem=52.12 m free gpu memory: [1228] iter= 600, img_loss=0.068432, reg_loss=0.220145, lr=0.01516, time=331.6 ms, rem=51.95 m free gpu memory: [1228] iter= 610, img_loss=0.046744, reg_loss=0.218927, lr=0.01509, time=332.9 ms, rem=52.10 m free gpu memory: [1228] iter= 620, img_loss=0.095495, reg_loss=0.217773, lr=0.01503, time=334.2 ms, rem=52.25 m free gpu memory: [1228] iter= 630, img_loss=0.083856, reg_loss=0.216643, lr=0.01496, time=334.2 ms, rem=52.19 m free gpu memory: [1228] iter= 640, img_loss=0.059069, reg_loss=0.215472, lr=0.01489, time=331.8 ms, rem=51.76 m free gpu memory: [1229] iter= 650, img_loss=0.038091, reg_loss=0.214378, lr=0.01482, time=332.1 ms, rem=51.75 m free gpu memory: [1229] iter= 660, img_loss=0.065835, reg_loss=0.213510, lr=0.01475, time=332.5 ms, rem=51.76 m free gpu memory: [1229] iter= 670, img_loss=0.015726, reg_loss=0.212773, lr=0.01468, time=331.8 ms, rem=51.60 m free gpu memory: [1229] iter= 680, img_loss=0.030530, reg_loss=0.211792, lr=0.01462, time=343.7 ms, rem=53.39 m free gpu memory: [1220] iter= 690, img_loss=0.015212, reg_loss=0.210694, lr=0.01455, time=335.7 ms, rem=52.09 m free gpu memory: [1220] iter= 700, img_loss=0.077175, reg_loss=0.209571, lr=0.01448, time=338.7 ms, rem=52.50 m free gpu memory: [1229] iter= 710, img_loss=0.031941, reg_loss=0.208392, lr=0.01442, time=353.1 ms, rem=54.67 m free gpu memory: [1220] iter= 720, img_loss=0.042972, reg_loss=0.207231, lr=0.01435, time=334.6 ms, rem=51.75 m free gpu memory: [1220] iter= 730, img_loss=0.042593, reg_loss=0.206098, lr=0.01428, time=347.2 ms, rem=53.64 m free gpu memory: [1229] iter= 740, img_loss=0.029113, reg_loss=0.204926, lr=0.01422, time=330.5 ms, rem=51.01 m free gpu memory: [1229] iter= 750, img_loss=0.061652, reg_loss=0.203847, lr=0.01415, time=332.6 ms, rem=51.28 m free gpu memory: [1229] iter= 760, img_loss=0.089841, reg_loss=0.202766, lr=0.01409, time=329.6 ms, rem=50.76 m free gpu memory: [1229] iter= 770, img_loss=0.035615, reg_loss=0.201626, lr=0.01402, time=332.1 ms, rem=51.09 m free gpu memory: [1229] iter= 780, img_loss=0.060848, reg_loss=0.200697, lr=0.01396, time=332.5 ms, rem=51.09 m free gpu memory: [1229] iter= 790, img_loss=0.030288, reg_loss=0.199766, lr=0.01389, time=330.4 ms, rem=50.72 m free gpu memory: [1229] iter= 800, img_loss=0.061218, reg_loss=0.198755, lr=0.01383, time=334.2 ms, rem=51.24 m free gpu memory: [1229] iter= 810, img_loss=0.040417, reg_loss=0.197670, lr=0.01377, time=335.8 ms, rem=51.43 m free gpu memory: [1229] iter= 820, img_loss=0.051640, reg_loss=0.196562, lr=0.01370, time=341.4 ms, rem=52.24 m free gpu memory: [1229] iter= 830, img_loss=0.075490, reg_loss=0.195512, lr=0.01364, time=349.0 ms, rem=53.34 m free gpu memory: [1220] iter= 840, img_loss=0.058173, reg_loss=0.194271, lr=0.01358, time=333.5 ms, rem=50.91 m free gpu memory: [1220] iter= 850, img_loss=0.072006, reg_loss=0.193277, lr=0.01352, time=340.6 ms, rem=51.94 m free gpu memory: [1220] iter= 860, img_loss=0.046849, reg_loss=0.192350, lr=0.01345, time=334.8 ms, rem=51.00 m free gpu memory: [1229] iter= 870, img_loss=0.078958, reg_loss=0.191257, lr=0.01339, time=343.6 ms, rem=52.29 m free gpu memory: [1229] iter= 880, img_loss=0.053510, reg_loss=0.190210, lr=0.01333, time=332.2 ms, rem=50.49 m free gpu memory: [1229] iter= 890, img_loss=0.031054, reg_loss=0.189208, lr=0.01327, time=332.6 ms, rem=50.50 m free gpu memory: [1229] iter= 900, img_loss=0.032931, reg_loss=0.188170, lr=0.01321, time=331.5 ms, rem=50.28 m free gpu memory: [1229] iter= 910, img_loss=0.034421, reg_loss=0.187112, lr=0.01315, time=351.3 ms, rem=53.22 m free gpu memory: [1216] iter= 920, img_loss=0.024211, reg_loss=0.186034, lr=0.01309, time=336.9 ms, rem=50.99 m free gpu memory: [1225] iter= 930, img_loss=0.062483, reg_loss=0.184821, lr=0.01303, time=348.7 ms, rem=52.71 m free gpu memory: [1223] iter= 940, img_loss=0.025642, reg_loss=0.183682, lr=0.01297, time=330.1 ms, rem=49.85 m free gpu memory: [1223] iter= 950, img_loss=0.030485, reg_loss=0.182665, lr=0.01291, time=330.5 ms, rem=49.85 m free gpu memory: [1223] iter= 960, img_loss=0.041848, reg_loss=0.181569, lr=0.01285, time=330.7 ms, rem=49.83 m free gpu memory: [1223] iter= 970, img_loss=0.069039, reg_loss=0.180350, lr=0.01279, time=330.1 ms, rem=49.68 m free gpu memory: [1223] iter= 980, img_loss=0.036390, reg_loss=0.179175, lr=0.01273, time=329.6 ms, rem=49.55 m free gpu memory: [1223] iter= 990, img_loss=0.031682, reg_loss=0.178091, lr=0.01267, time=328.0 ms, rem=49.25 m free gpu memory: [1223] iter= 1000, img_loss=0.043526, reg_loss=0.176997, lr=0.01261, time=334.1 ms, rem=50.12 m free gpu memory: [1223] iter= 1010, img_loss=0.038272, reg_loss=0.175888, lr=0.01256, time=330.3 ms, rem=49.49 m free gpu memory: [1083] iter= 1020, img_loss=0.035919, reg_loss=0.174753, lr=0.01250, time=332.0 ms, rem=49.69 m free gpu memory: [1083] iter= 1030, img_loss=0.036843, reg_loss=0.173719, lr=0.01244, time=330.3 ms, rem=49.38 m free gpu memory: [1083] iter= 1040, img_loss=0.014721, reg_loss=0.172675, lr=0.01238, time=329.3 ms, rem=49.18 m free gpu memory: [1083] iter= 1050, img_loss=0.034947, reg_loss=0.171596, lr=0.01233, time=330.1 ms, rem=49.24 m free gpu memory: [1083] iter= 1060, img_loss=0.105641, reg_loss=0.170422, lr=0.01227, time=329.8 ms, rem=49.14 m free gpu memory: [1083] iter= 1070, img_loss=0.079758, reg_loss=0.169244, lr=0.01221, time=330.7 ms, rem=49.22 m free gpu memory: [1083] iter= 1080, img_loss=0.029878, reg_loss=0.168198, lr=0.01216, time=335.1 ms, rem=49.82 m free gpu memory: [1083] iter= 1090, img_loss=0.047070, reg_loss=0.167179, lr=0.01210, time=346.6 ms, rem=51.47 m free gpu memory: [1075] iter= 1100, img_loss=0.040635, reg_loss=0.166114, lr=0.01205, time=333.8 ms, rem=49.51 m free gpu memory: [1075] iter= 1110, img_loss=0.060730, reg_loss=0.165000, lr=0.01199, time=334.0 ms, rem=49.49 m free gpu memory: [1075] iter= 1120, img_loss=0.076433, reg_loss=0.163874, lr=0.01194, time=344.2 ms, rem=50.94 m free gpu memory: [1083] iter= 1130, img_loss=0.067390, reg_loss=0.162706, lr=0.01188, time=348.0 ms, rem=51.45 m free gpu memory: [1085] iter= 1140, img_loss=0.037461, reg_loss=0.161732, lr=0.01183, time=332.1 ms, rem=49.04 m free gpu memory: [1085] iter= 1150, img_loss=0.052903, reg_loss=0.160677, lr=0.01177, time=332.5 ms, rem=49.04 m free gpu memory: [1085] iter= 1160, img_loss=0.047440, reg_loss=0.159534, lr=0.01172, time=334.7 ms, rem=49.31 m free gpu memory: [1085] iter= 1170, img_loss=0.077329, reg_loss=0.158440, lr=0.01166, time=331.2 ms, rem=48.74 m free gpu memory: [1085] iter= 1180, img_loss=0.078900, reg_loss=0.157336, lr=0.01161, time=331.9 ms, rem=48.79 m free gpu memory: [1085] iter= 1190, img_loss=0.037164, reg_loss=0.156209, lr=0.01156, time=330.4 ms, rem=48.51 m free gpu memory: [1085] iter= 1200, img_loss=0.107620, reg_loss=0.155183, lr=0.01150, time=332.4 ms, rem=48.75 m free gpu memory: [1085] iter= 1210, img_loss=0.089469, reg_loss=0.154239, lr=0.01145, time=331.5 ms, rem=48.57 m free gpu memory: [1085] iter= 1220, img_loss=0.066752, reg_loss=0.153374, lr=0.01140, time=330.4 ms, rem=48.35 m free gpu memory: [1085] iter= 1230, img_loss=0.057211, reg_loss=0.152384, lr=0.01135, time=330.2 ms, rem=48.26 m free gpu memory: [1085] iter= 1240, img_loss=0.025078, reg_loss=0.151295, lr=0.01129, time=329.4 ms, rem=48.09 m free gpu memory: [1085] iter= 1250, img_loss=0.073495, reg_loss=0.150231, lr=0.01124, time=331.7 ms, rem=48.37 m free gpu memory: [1085] iter= 1260, img_loss=0.043161, reg_loss=0.149105, lr=0.01119, time=328.6 ms, rem=47.87 m free gpu memory: [1085] iter= 1270, img_loss=0.054619, reg_loss=0.147938, lr=0.01114, time=331.0 ms, rem=48.16 m free gpu memory: [1085] iter= 1280, img_loss=0.066879, reg_loss=0.146806, lr=0.01109, time=329.5 ms, rem=47.89 m free gpu memory: [1085] iter= 1290, img_loss=0.058439, reg_loss=0.145683, lr=0.01104, time=329.6 ms, rem=47.85 m free gpu memory: [1085] iter= 1300, img_loss=0.069395, reg_loss=0.144538, lr=0.01099, time=329.5 ms, rem=47.78 m free gpu memory: [1085] iter= 1310, img_loss=0.038660, reg_loss=0.143423, lr=0.01094, time=331.3 ms, rem=47.98 m free gpu memory: [1085] iter= 1320, img_loss=0.059340, reg_loss=0.142368, lr=0.01089, time=330.6 ms, rem=47.83 m free gpu memory: [1085] iter= 1330, img_loss=0.058031, reg_loss=0.141347, lr=0.01084, time=330.3 ms, rem=47.73 m free gpu memory: [1085] iter= 1340, img_loss=0.043750, reg_loss=0.140296, lr=0.01079, time=329.5 ms, rem=47.56 m free gpu memory: [1085] iter= 1350, img_loss=0.061712, reg_loss=0.139194, lr=0.01074, time=331.6 ms, rem=47.81 m free gpu memory: [1085] iter= 1360, img_loss=0.054764, reg_loss=0.138080, lr=0.01069, time=329.8 ms, rem=47.49 m free gpu memory: [1085] iter= 1370, img_loss=0.052170, reg_loss=0.136942, lr=0.01064, time=331.1 ms, rem=47.62 m free gpu memory: [1085] iter= 1380, img_loss=0.048567, reg_loss=0.135844, lr=0.01059, time=331.1 ms, rem=47.57 m free gpu memory: [1085] iter= 1390, img_loss=0.013059, reg_loss=0.134807, lr=0.01054, time=330.9 ms, rem=47.48 m free gpu memory: [1085] iter= 1400, img_loss=0.033507, reg_loss=0.133769, lr=0.01049, time=331.4 ms, rem=47.50 m free gpu memory: [1085] iter= 1410, img_loss=0.033951, reg_loss=0.132731, lr=0.01044, time=331.3 ms, rem=47.43 m free gpu memory: [1085] iter= 1420, img_loss=0.044250, reg_loss=0.131655, lr=0.01040, time=332.7 ms, rem=47.58 m free gpu memory: [1085] iter= 1430, img_loss=0.053107, reg_loss=0.130513, lr=0.01035, time=329.6 ms, rem=47.08 m free gpu memory: [1085] iter= 1440, img_loss=0.025770, reg_loss=0.129445, lr=0.01030, time=328.4 ms, rem=46.85 m free gpu memory: [1085] iter= 1450, img_loss=0.039279, reg_loss=0.128354, lr=0.01025, time=328.8 ms, rem=46.85 m free gpu memory: [1085] iter= 1460, img_loss=0.059036, reg_loss=0.127219, lr=0.01021, time=330.1 ms, rem=46.98 m free gpu memory: [1085] iter= 1470, img_loss=0.059612, reg_loss=0.126104, lr=0.01016, time=329.1 ms, rem=46.79 m free gpu memory: [1085] iter= 1480, img_loss=0.041292, reg_loss=0.125006, lr=0.01011, time=328.4 ms, rem=46.63 m free gpu memory: [1085] iter= 1490, img_loss=0.084344, reg_loss=0.123913, lr=0.01007, time=330.6 ms, rem=46.89 m free gpu memory: [1085] iter= 1500, img_loss=0.053639, reg_loss=0.122834, lr=0.01002, time=331.7 ms, rem=46.99 m free gpu memory: [1085] iter= 1510, img_loss=0.067543, reg_loss=0.121729, lr=0.00997, time=331.1 ms, rem=46.85 m free gpu memory: [945] iter= 1520, img_loss=0.070325, reg_loss=0.120632, lr=0.00993, time=330.8 ms, rem=46.75 m free gpu memory: [945] iter= 1530, img_loss=0.031384, reg_loss=0.119591, lr=0.00988, time=329.8 ms, rem=46.56 m free gpu memory: [945] iter= 1540, img_loss=0.088435, reg_loss=0.118540, lr=0.00984, time=333.3 ms, rem=47.00 m free gpu memory: [945] iter= 1550, img_loss=0.068972, reg_loss=0.117477, lr=0.00979, time=330.2 ms, rem=46.50 m free gpu memory: [945] iter= 1560, img_loss=0.058348, reg_loss=0.116359, lr=0.00975, time=331.4 ms, rem=46.62 m free gpu memory: [945] iter= 1570, img_loss=0.066280, reg_loss=0.115254, lr=0.00970, time=334.0 ms, rem=46.93 m free gpu memory: [945] iter= 1580, img_loss=0.046649, reg_loss=0.114272, lr=0.00966, time=329.7 ms, rem=46.27 m free gpu memory: [945] iter= 1590, img_loss=0.041926, reg_loss=0.113224, lr=0.00961, time=331.0 ms, rem=46.40 m free gpu memory: [945] iter= 1600, img_loss=0.035020, reg_loss=0.112164, lr=0.00957, time=330.6 ms, rem=46.28 m free gpu memory: [945] iter= 1610, img_loss=0.052143, reg_loss=0.111085, lr=0.00952, time=332.8 ms, rem=46.54 m free gpu memory: [945] iter= 1620, img_loss=0.048569, reg_loss=0.110044, lr=0.00948, time=330.7 ms, rem=46.19 m free gpu memory: [945] iter= 1630, img_loss=0.032927, reg_loss=0.108940, lr=0.00944, time=329.9 ms, rem=46.02 m free gpu memory: [945] iter= 1640, img_loss=0.040428, reg_loss=0.107814, lr=0.00939, time=330.0 ms, rem=45.98 m free gpu memory: [945] iter= 1650, img_loss=0.051104, reg_loss=0.106675, lr=0.00935, time=332.9 ms, rem=46.33 m free gpu memory: [945] iter= 1660, img_loss=0.065559, reg_loss=0.105586, lr=0.00931, time=331.6 ms, rem=46.09 m free gpu memory: [945] iter= 1670, img_loss=0.043200, reg_loss=0.104498, lr=0.00926, time=329.9 ms, rem=45.80 m free gpu memory: [945] iter= 1680, img_loss=0.033310, reg_loss=0.103416, lr=0.00922, time=330.6 ms, rem=45.84 m free gpu memory: [945] iter= 1690, img_loss=0.039434, reg_loss=0.102356, lr=0.00918, time=330.3 ms, rem=45.75 m free gpu memory: [945] iter= 1700, img_loss=0.051716, reg_loss=0.101289, lr=0.00914, time=329.7 ms, rem=45.61 m free gpu memory: [945] iter= 1710, img_loss=0.034509, reg_loss=0.100188, lr=0.00910, time=330.6 ms, rem=45.68 m free gpu memory: [945] iter= 1720, img_loss=0.056750, reg_loss=0.099080, lr=0.00905, time=331.1 ms, rem=45.69 m free gpu memory: [945] iter= 1730, img_loss=0.024530, reg_loss=0.098020, lr=0.00901, time=330.4 ms, rem=45.54 m free gpu memory: [945] iter= 1740, img_loss=0.032164, reg_loss=0.096967, lr=0.00897, time=330.7 ms, rem=45.53 m free gpu memory: [945] iter= 1750, img_loss=0.033079, reg_loss=0.095904, lr=0.00893, time=330.3 ms, rem=45.42 m free gpu memory: [945] iter= 1760, img_loss=0.042827, reg_loss=0.094802, lr=0.00889, time=331.4 ms, rem=45.51 m free gpu memory: [945] iter= 1770, img_loss=0.048389, reg_loss=0.093672, lr=0.00885, time=330.8 ms, rem=45.37 m free gpu memory: [945] iter= 1780, img_loss=0.053627, reg_loss=0.092551, lr=0.00881, time=333.9 ms, rem=45.74 m free gpu memory: [945] iter= 1790, img_loss=0.057405, reg_loss=0.091483, lr=0.00877, time=331.5 ms, rem=45.36 m free gpu memory: [945] iter= 1800, img_loss=0.082753, reg_loss=0.090410, lr=0.00873, time=332.6 ms, rem=45.46 m free gpu memory: [945] iter= 1810, img_loss=0.041817, reg_loss=0.089354, lr=0.00869, time=330.2 ms, rem=45.07 m free gpu memory: [945] iter= 1820, img_loss=0.054646, reg_loss=0.088337, lr=0.00865, time=330.5 ms, rem=45.06 m free gpu memory: [945] iter= 1830, img_loss=0.022355, reg_loss=0.087263, lr=0.00861, time=329.6 ms, rem=44.88 m free gpu memory: [945] iter= 1840, img_loss=0.043761, reg_loss=0.086187, lr=0.00857, time=331.2 ms, rem=45.04 m free gpu memory: [945] iter= 1850, img_loss=0.073043, reg_loss=0.085096, lr=0.00853, time=331.2 ms, rem=44.99 m free gpu memory: [945] iter= 1860, img_loss=0.048383, reg_loss=0.083995, lr=0.00849, time=329.9 ms, rem=44.76 m free gpu memory: [945] iter= 1870, img_loss=0.039671, reg_loss=0.082937, lr=0.00845, time=328.8 ms, rem=44.55 m free gpu memory: [945] iter= 1880, img_loss=0.041778, reg_loss=0.081874, lr=0.00841, time=330.1 ms, rem=44.67 m free gpu memory: [945] iter= 1890, img_loss=0.045670, reg_loss=0.080809, lr=0.00837, time=331.1 ms, rem=44.75 m free gpu memory: [945] iter= 1900, img_loss=0.060370, reg_loss=0.079721, lr=0.00833, time=330.6 ms, rem=44.63 m free gpu memory: [945] iter= 1910, img_loss=0.089386, reg_loss=0.078620, lr=0.00830, time=334.3 ms, rem=45.08 m free gpu memory: [805] iter= 1920, img_loss=0.047037, reg_loss=0.077543, lr=0.00826, time=329.6 ms, rem=44.39 m free gpu memory: [805] iter= 1930, img_loss=0.064136, reg_loss=0.076463, lr=0.00822, time=331.2 ms, rem=44.55 m free gpu memory: [805] iter= 1940, img_loss=0.013530, reg_loss=0.075400, lr=0.00818, time=328.6 ms, rem=44.14 m free gpu memory: [805] iter= 1950, img_loss=0.024091, reg_loss=0.074334, lr=0.00814, time=329.2 ms, rem=44.17 m free gpu memory: [805] iter= 1960, img_loss=0.057126, reg_loss=0.073267, lr=0.00811, time=330.6 ms, rem=44.30 m free gpu memory: [805] iter= 1970, img_loss=0.034115, reg_loss=0.072202, lr=0.00807, time=330.2 ms, rem=44.19 m free gpu memory: [805] iter= 1980, img_loss=0.020793, reg_loss=0.071131, lr=0.00803, time=329.7 ms, rem=44.07 m free gpu memory: [805] iter= 1990, img_loss=0.041585, reg_loss=0.070078, lr=0.00800, time=329.7 ms, rem=44.01 m free gpu memory: [805] iter= 2000, img_loss=0.052488, reg_loss=0.069002, lr=0.00796, time=329.5 ms, rem=43.93 m free gpu memory: [805] iter= 2010, img_loss=0.044785, reg_loss=0.067931, lr=0.00792, time=330.6 ms, rem=44.03 m free gpu memory: [805] iter= 2020, img_loss=0.020746, reg_loss=0.066858, lr=0.00789, time=329.7 ms, rem=43.85 m free gpu memory: [805] iter= 2030, img_loss=0.090321, reg_loss=0.065762, lr=0.00785, time=333.1 ms, rem=44.25 m free gpu memory: [805] iter= 2040, img_loss=0.058465, reg_loss=0.064664, lr=0.00781, time=330.4 ms, rem=43.83 m free gpu memory: [805] iter= 2050, img_loss=0.068358, reg_loss=0.063557, lr=0.00778, time=330.5 ms, rem=43.79 m free gpu memory: [805] iter= 2060, img_loss=0.055065, reg_loss=0.062474, lr=0.00774, time=330.5 ms, rem=43.74 m free gpu memory: [805] iter= 2070, img_loss=0.052013, reg_loss=0.061399, lr=0.00771, time=331.4 ms, rem=43.80 m free gpu memory: [805] iter= 2080, img_loss=0.052451, reg_loss=0.060325, lr=0.00767, time=332.5 ms, rem=43.89 m free gpu memory: [805] iter= 2090, img_loss=0.040532, reg_loss=0.059250, lr=0.00764, time=329.5 ms, rem=43.44 m free gpu memory: [805] iter= 2100, img_loss=0.053650, reg_loss=0.058172, lr=0.00760, time=332.6 ms, rem=43.79 m free gpu memory: [805] iter= 2110, img_loss=0.028922, reg_loss=0.057095, lr=0.00757, time=332.5 ms, rem=43.72 m free gpu memory: [805] iter= 2120, img_loss=0.032442, reg_loss=0.056022, lr=0.00753, time=329.2 ms, rem=43.23 m free gpu memory: [805] iter= 2130, img_loss=0.037381, reg_loss=0.054952, lr=0.00750, time=329.6 ms, rem=43.23 m free gpu memory: [805] iter= 2140, img_loss=0.021378, reg_loss=0.053862, lr=0.00746, time=330.1 ms, rem=43.24 m free gpu memory: [805] iter= 2150, img_loss=0.057829, reg_loss=0.052773, lr=0.00743, time=331.2 ms, rem=43.33 m free gpu memory: [805] iter= 2160, img_loss=0.027015, reg_loss=0.051714, lr=0.00739, time=328.7 ms, rem=42.95 m free gpu memory: [805] iter= 2170, img_loss=0.047624, reg_loss=0.050618, lr=0.00736, time=330.7 ms, rem=43.16 m free gpu memory: [805] iter= 2180, img_loss=0.059408, reg_loss=0.049536, lr=0.00733, time=330.2 ms, rem=43.04 m free gpu memory: [805] iter= 2190, img_loss=0.051679, reg_loss=0.048458, lr=0.00729, time=329.8 ms, rem=42.93 m free gpu memory: [805] iter= 2200, img_loss=0.042705, reg_loss=0.047373, lr=0.00726, time=331.0 ms, rem=43.03 m free gpu memory: [805] iter= 2210, img_loss=0.049412, reg_loss=0.046293, lr=0.00722, time=329.7 ms, rem=42.81 m free gpu memory: [805] iter= 2220, img_loss=0.058051, reg_loss=0.045194, lr=0.00719, time=330.5 ms, rem=42.85 m free gpu memory: [805] iter= 2230, img_loss=0.061793, reg_loss=0.044107, lr=0.00716, time=330.3 ms, rem=42.77 m free gpu memory: [805] iter= 2240, img_loss=0.052723, reg_loss=0.043039, lr=0.00713, time=329.4 ms, rem=42.60 m free gpu memory: [805] iter= 2250, img_loss=0.039994, reg_loss=0.041972, lr=0.00709, time=328.7 ms, rem=42.46 m free gpu memory: [805] iter= 2260, img_loss=0.049762, reg_loss=0.040900, lr=0.00706, time=330.4 ms, rem=42.62 m free gpu memory: [805] iter= 2270, img_loss=0.051946, reg_loss=0.039826, lr=0.00703, time=330.2 ms, rem=42.54 m free gpu memory: [805] iter= 2280, img_loss=0.048347, reg_loss=0.038740, lr=0.00700, time=331.0 ms, rem=42.59 m free gpu memory: [805] iter= 2290, img_loss=0.076316, reg_loss=0.037639, lr=0.00696, time=331.3 ms, rem=42.57 m free gpu memory: [805] iter= 2300, img_loss=0.029078, reg_loss=0.036557, lr=0.00693, time=329.3 ms, rem=42.26 m free gpu memory: [805] iter= 2310, img_loss=0.032390, reg_loss=0.035486, lr=0.00690, time=327.9 ms, rem=42.03 m free gpu memory: [665] iter= 2320, img_loss=0.033963, reg_loss=0.034398, lr=0.00687, time=330.3 ms, rem=42.28 m free gpu memory: [665] iter= 2330, img_loss=0.035381, reg_loss=0.033333, lr=0.00684, time=328.4 ms, rem=41.98 m free gpu memory: [665] iter= 2340, img_loss=0.036217, reg_loss=0.032241, lr=0.00681, time=329.5 ms, rem=42.07 m free gpu memory: [665] iter= 2350, img_loss=0.044837, reg_loss=0.031153, lr=0.00677, time=329.7 ms, rem=42.04 m free gpu memory: [665] iter= 2360, img_loss=0.025021, reg_loss=0.030069, lr=0.00674, time=330.1 ms, rem=42.03 m free gpu memory: [665] iter= 2370, img_loss=0.021576, reg_loss=0.028989, lr=0.00671, time=329.6 ms, rem=41.91 m free gpu memory: [665] iter= 2380, img_loss=0.039770, reg_loss=0.027907, lr=0.00668, time=328.1 ms, rem=41.67 m free gpu memory: [665] iter= 2390, img_loss=0.046784, reg_loss=0.026816, lr=0.00665, time=330.5 ms, rem=41.92 m free gpu memory: [665] iter= 2400, img_loss=0.026784, reg_loss=0.025740, lr=0.00662, time=329.5 ms, rem=41.74 m free gpu memory: [665] iter= 2410, img_loss=0.066329, reg_loss=0.024656, lr=0.00659, time=331.3 ms, rem=41.91 m free gpu memory: [665] iter= 2420, img_loss=0.029290, reg_loss=0.023581, lr=0.00656, time=330.4 ms, rem=41.74 m free gpu memory: [665] iter= 2430, img_loss=0.041557, reg_loss=0.022517, lr=0.00653, time=329.4 ms, rem=41.56 m free gpu memory: [665] iter= 2440, img_loss=0.020394, reg_loss=0.021429, lr=0.00650, time=327.1 ms, rem=41.21 m free gpu memory: [665] iter= 2450, img_loss=0.054473, reg_loss=0.020340, lr=0.00647, time=330.8 ms, rem=41.63 m free gpu memory: [665] iter= 2460, img_loss=0.054478, reg_loss=0.019251, lr=0.00644, time=331.1 ms, rem=41.61 m free gpu memory: [665] iter= 2470, img_loss=0.040699, reg_loss=0.018172, lr=0.00641, time=330.2 ms, rem=41.44 m free gpu memory: [665] iter= 2480, img_loss=0.069630, reg_loss=0.017080, lr=0.00638, time=330.7 ms, rem=41.45 m free gpu memory: [665] iter= 2490, img_loss=0.073474, reg_loss=0.015999, lr=0.00635, time=331.3 ms, rem=41.47 m free gpu memory: [665] iter= 2500, img_loss=0.033118, reg_loss=0.014920, lr=0.00632, time=330.4 ms, rem=41.30 m free gpu memory: [665] iter= 2510, img_loss=0.071387, reg_loss=0.014444, lr=0.00629, time=328.3 ms, rem=40.98 m free gpu memory: [665] iter= 2520, img_loss=0.070495, reg_loss=0.014433, lr=0.00626, time=330.7 ms, rem=41.23 m free gpu memory: [665] iter= 2530, img_loss=0.017345, reg_loss=0.014445, lr=0.00623, time=329.3 ms, rem=41.00 m free gpu memory: [665] iter= 2540, img_loss=0.030358, reg_loss=0.014438, lr=0.00621, time=329.3 ms, rem=40.94 m free gpu memory: [665] iter= 2550, img_loss=0.074092, reg_loss=0.014433, lr=0.00618, time=328.5 ms, rem=40.79 m free gpu memory: [665] iter= 2560, img_loss=0.049920, reg_loss=0.014434, lr=0.00615, time=330.8 ms, rem=41.02 m free gpu memory: [665] iter= 2570, img_loss=0.056288, reg_loss=0.014429, lr=0.00612, time=329.8 ms, rem=40.84 m free gpu memory: [665] iter= 2580, img_loss=0.076544, reg_loss=0.014432, lr=0.00609, time=329.9 ms, rem=40.80 m free gpu memory: [665] iter= 2590, img_loss=0.020510, reg_loss=0.014441, lr=0.00606, time=330.8 ms, rem=40.85 m free gpu memory: [665] iter= 2600, img_loss=0.044505, reg_loss=0.014453, lr=0.00604, time=328.7 ms, rem=40.54 m free gpu memory: [665] iter= 2610, img_loss=0.036814, reg_loss=0.014451, lr=0.00601, time=331.0 ms, rem=40.77 m free gpu memory: [665] iter= 2620, img_loss=0.043905, reg_loss=0.014438, lr=0.00598, time=332.4 ms, rem=40.89 m free gpu memory: [665] iter= 2630, img_loss=0.041876, reg_loss=0.014444, lr=0.00595, time=329.9 ms, rem=40.52 m free gpu memory: [665] iter= 2640, img_loss=0.083181, reg_loss=0.014451, lr=0.00593, time=328.8 ms, rem=40.33 m free gpu memory: [665] iter= 2650, img_loss=0.047746, reg_loss=0.014432, lr=0.00590, time=331.4 ms, rem=40.60 m free gpu memory: [665] iter= 2660, img_loss=0.049818, reg_loss=0.014437, lr=0.00587, time=330.2 ms, rem=40.39 m free gpu memory: [665] iter= 2670, img_loss=0.042876, reg_loss=0.014443, lr=0.00585, time=330.5 ms, rem=40.38 m free gpu memory: [665] iter= 2680, img_loss=0.054329, reg_loss=0.014446, lr=0.00582, time=330.1 ms, rem=40.27 m free gpu memory: [665] iter= 2690, img_loss=0.043783, reg_loss=0.014457, lr=0.00579, time=328.6 ms, rem=40.03 m free gpu memory: [665] iter= 2700, img_loss=0.027600, reg_loss=0.014457, lr=0.00577, time=328.3 ms, rem=39.94 m free gpu memory: [665] iter= 2710, img_loss=0.043856, reg_loss=0.014469, lr=0.00574, time=331.2 ms, rem=40.24 m free gpu memory: [665] iter= 2720, img_loss=0.034602, reg_loss=0.014468, lr=0.00571, time=329.0 ms, rem=39.92 m free gpu memory: [665] iter= 2730, img_loss=0.016815, reg_loss=0.014463, lr=0.00569, time=330.5 ms, rem=40.05 m free gpu memory: [665] iter= 2740, img_loss=0.052514, reg_loss=0.014456, lr=0.00566, time=330.2 ms, rem=39.95 m free gpu memory: [665] iter= 2750, img_loss=0.040681, reg_loss=0.014457, lr=0.00563, time=330.0 ms, rem=39.88 m free gpu memory: [665] iter= 2760, img_loss=0.045532, reg_loss=0.014454, lr=0.00561, time=330.1 ms, rem=39.83 m free gpu memory: [665] iter= 2770, img_loss=0.053692, reg_loss=0.014459, lr=0.00558, time=329.3 ms, rem=39.68 m free gpu memory: [665] iter= 2780, img_loss=0.044671, reg_loss=0.014456, lr=0.00556, time=329.2 ms, rem=39.61 m free gpu memory: [665] iter= 2790, img_loss=0.052636, reg_loss=0.014459, lr=0.00553, time=330.7 ms, rem=39.74 m free gpu memory: [525] iter= 2800, img_loss=0.081569, reg_loss=0.014461, lr=0.00551, time=330.2 ms, rem=39.62 m free gpu memory: [525] iter= 2810, img_loss=0.022860, reg_loss=0.014469, lr=0.00548, time=329.4 ms, rem=39.47 m free gpu memory: [525] iter= 2820, img_loss=0.034036, reg_loss=0.014471, lr=0.00546, time=329.9 ms, rem=39.48 m free gpu memory: [525] iter= 2830, img_loss=0.040181, reg_loss=0.014462, lr=0.00543, time=331.0 ms, rem=39.55 m free gpu memory: [525] iter= 2840, img_loss=0.036876, reg_loss=0.014471, lr=0.00541, time=327.9 ms, rem=39.13 m free gpu memory: [525] iter= 2850, img_loss=0.015517, reg_loss=0.014478, lr=0.00538, time=328.1 ms, rem=39.10 m free gpu memory: [525] iter= 2860, img_loss=0.019055, reg_loss=0.014472, lr=0.00536, time=329.0 ms, rem=39.15 m free gpu memory: [525] iter= 2870, img_loss=0.035120, reg_loss=0.014466, lr=0.00533, time=328.7 ms, rem=39.06 m free gpu memory: [525] iter= 2880, img_loss=0.035915, reg_loss=0.014465, lr=0.00531, time=330.1 ms, rem=39.17 m free gpu memory: [525] iter= 2890, img_loss=0.057002, reg_loss=0.014466, lr=0.00528, time=331.0 ms, rem=39.22 m free gpu memory: [525] iter= 2900, img_loss=0.030073, reg_loss=0.014475, lr=0.00526, time=328.4 ms, rem=38.86 m free gpu memory: [525] iter= 2910, img_loss=0.041711, reg_loss=0.014463, lr=0.00523, time=330.3 ms, rem=39.03 m free gpu memory: [525] iter= 2920, img_loss=0.031049, reg_loss=0.014474, lr=0.00521, time=327.8 ms, rem=38.68 m free gpu memory: [525] iter= 2930, img_loss=0.046165, reg_loss=0.014470, lr=0.00519, time=328.3 ms, rem=38.68 m free gpu memory: [525] iter= 2940, img_loss=0.057015, reg_loss=0.014458, lr=0.00516, time=330.4 ms, rem=38.88 m free gpu memory: [525] iter= 2950, img_loss=0.068843, reg_loss=0.014463, lr=0.00514, time=328.8 ms, rem=38.63 m free gpu memory: [525] iter= 2960, img_loss=0.027196, reg_loss=0.014451, lr=0.00511, time=330.4 ms, rem=38.77 m free gpu memory: [525] iter= 2970, img_loss=0.043205, reg_loss=0.014468, lr=0.00509, time=329.1 ms, rem=38.56 m free gpu memory: [525] iter= 2980, img_loss=0.063256, reg_loss=0.014453, lr=0.00507, time=330.1 ms, rem=38.62 m free gpu memory: [525] iter= 2990, img_loss=0.027287, reg_loss=0.014461, lr=0.00504, time=329.4 ms, rem=38.48 m free gpu memory: [525] iter= 3000, img_loss=0.065564, reg_loss=0.014458, lr=0.00502, time=331.3 ms, rem=38.65 m free gpu memory: [525] iter= 3010, img_loss=0.073006, reg_loss=0.014458, lr=0.00500, time=330.7 ms, rem=38.53 m free gpu memory: [525] iter= 3020, img_loss=0.025216, reg_loss=0.014463, lr=0.00498, time=329.9 ms, rem=38.38 m free gpu memory: [525] iter= 3030, img_loss=0.053702, reg_loss=0.014471, lr=0.00495, time=329.7 ms, rem=38.30 m free gpu memory: [525] iter= 3040, img_loss=0.070900, reg_loss=0.014463, lr=0.00493, time=331.9 ms, rem=38.50 m free gpu memory: [525] iter= 3050, img_loss=0.072327, reg_loss=0.014466, lr=0.00491, time=330.2 ms, rem=38.25 m free gpu memory: [525] iter= 3060, img_loss=0.053629, reg_loss=0.014476, lr=0.00488, time=329.3 ms, rem=38.09 m free gpu memory: [525] iter= 3070, img_loss=0.058268, reg_loss=0.014466, lr=0.00486, time=330.5 ms, rem=38.17 m free gpu memory: [525] iter= 3080, img_loss=0.065430, reg_loss=0.014460, lr=0.00484, time=330.6 ms, rem=38.13 m free gpu memory: [525] iter= 3090, img_loss=0.042566, reg_loss=0.014464, lr=0.00482, time=331.9 ms, rem=38.22 m free gpu memory: [525] iter= 3100, img_loss=0.028584, reg_loss=0.014469, lr=0.00480, time=331.4 ms, rem=38.11 m free gpu memory: [385] iter= 3110, img_loss=0.034170, reg_loss=0.014471, lr=0.00477, time=331.9 ms, rem=38.11 m free gpu memory: [385] iter= 3120, img_loss=0.051107, reg_loss=0.014469, lr=0.00475, time=329.7 ms, rem=37.81 m free gpu memory: [385] iter= 3130, img_loss=0.054363, reg_loss=0.014458, lr=0.00473, time=330.1 ms, rem=37.80 m free gpu memory: [385] iter= 3140, img_loss=0.047216, reg_loss=0.014466, lr=0.00471, time=330.2 ms, rem=37.75 m free gpu memory: [385] iter= 3150, img_loss=0.027841, reg_loss=0.014487, lr=0.00469, time=329.2 ms, rem=37.58 m free gpu memory: [385] iter= 3160, img_loss=0.060935, reg_loss=0.014465, lr=0.00466, time=329.9 ms, rem=37.61 m free gpu memory: [385] iter= 3170, img_loss=0.023763, reg_loss=0.014469, lr=0.00464, time=330.8 ms, rem=37.66 m free gpu memory: [384] iter= 3180, img_loss=0.036356, reg_loss=0.014472, lr=0.00462, time=328.5 ms, rem=37.34 m free gpu memory: [384] iter= 3190, img_loss=0.027381, reg_loss=0.014476, lr=0.00460, time=328.4 ms, rem=37.27 m free gpu memory: [384] iter= 3200, img_loss=0.018379, reg_loss=0.014478, lr=0.00458, time=329.1 ms, rem=37.30 m free gpu memory: [384] iter= 3210, img_loss=0.051731, reg_loss=0.014476, lr=0.00456, time=332.1 ms, rem=37.58 m free gpu memory: [384] iter= 3220, img_loss=0.034294, reg_loss=0.014480, lr=0.00454, time=329.6 ms, rem=37.24 m free gpu memory: [384] iter= 3230, img_loss=0.063182, reg_loss=0.014476, lr=0.00452, time=330.8 ms, rem=37.33 m free gpu memory: [384] iter= 3240, img_loss=0.085643, reg_loss=0.014472, lr=0.00450, time=331.9 ms, rem=37.39 m free gpu memory: [384] iter= 3250, img_loss=0.021583, reg_loss=0.014474, lr=0.00448, time=329.1 ms, rem=37.02 m free gpu memory: [384] iter= 3260, img_loss=0.048556, reg_loss=0.014465, lr=0.00445, time=330.8 ms, rem=37.16 m free gpu memory: [384] iter= 3270, img_loss=0.048561, reg_loss=0.014457, lr=0.00443, time=332.2 ms, rem=37.26 m free gpu memory: [384] iter= 3280, img_loss=0.025289, reg_loss=0.014471, lr=0.00441, time=329.5 ms, rem=36.90 m free gpu memory: [384] iter= 3290, img_loss=0.038838, reg_loss=0.014448, lr=0.00439, time=332.3 ms, rem=37.16 m free gpu memory: [384] iter= 3300, img_loss=0.080321, reg_loss=0.014444, lr=0.00437, time=333.9 ms, rem=37.29 m free gpu memory: [384] iter= 3310, img_loss=0.026395, reg_loss=0.014464, lr=0.00435, time=331.1 ms, rem=36.92 m free gpu memory: [384] iter= 3320, img_loss=0.068453, reg_loss=0.014437, lr=0.00433, time=334.3 ms, rem=37.22 m free gpu memory: [384] iter= 3330, img_loss=0.067850, reg_loss=0.014445, lr=0.00431, time=331.2 ms, rem=36.82 m free gpu memory: [384] iter= 3340, img_loss=0.047755, reg_loss=0.014452, lr=0.00429, time=330.8 ms, rem=36.72 m free gpu memory: [384] iter= 3350, img_loss=0.059321, reg_loss=0.014446, lr=0.00427, time=332.3 ms, rem=36.83 m free gpu memory: [384] iter= 3360, img_loss=0.068542, reg_loss=0.014443, lr=0.00425, time=331.5 ms, rem=36.69 m free gpu memory: [384] iter= 3370, img_loss=0.059998, reg_loss=0.014452, lr=0.00423, time=328.7 ms, rem=36.32 m free gpu memory: [384] iter= 3380, img_loss=0.044663, reg_loss=0.014440, lr=0.00422, time=333.3 ms, rem=36.77 m free gpu memory: [384] iter= 3390, img_loss=0.038670, reg_loss=0.014456, lr=0.00420, time=331.4 ms, rem=36.51 m free gpu memory: [384] iter= 3400, img_loss=0.020869, reg_loss=0.014468, lr=0.00418, time=329.6 ms, rem=36.26 m free gpu memory: [384] iter= 3410, img_loss=0.058085, reg_loss=0.014461, lr=0.00416, time=332.9 ms, rem=36.56 m free gpu memory: [384] iter= 3420, img_loss=0.010722, reg_loss=0.014465, lr=0.00414, time=329.9 ms, rem=36.18 m free gpu memory: [384] iter= 3430, img_loss=0.073206, reg_loss=0.014468, lr=0.00412, time=330.2 ms, rem=36.16 m free gpu memory: [384] iter= 3440, img_loss=0.014885, reg_loss=0.014463, lr=0.00410, time=332.4 ms, rem=36.34 m free gpu memory: [384] iter= 3450, img_loss=0.052815, reg_loss=0.014463, lr=0.00408, time=330.2 ms, rem=36.05 m free gpu memory: [384] iter= 3460, img_loss=0.031899, reg_loss=0.014468, lr=0.00406, time=330.1 ms, rem=35.98 m free gpu memory: [384] iter= 3470, img_loss=0.046937, reg_loss=0.014464, lr=0.00404, time=331.5 ms, rem=36.08 m free gpu memory: [384] iter= 3480, img_loss=0.047084, reg_loss=0.014461, lr=0.00403, time=330.4 ms, rem=35.90 m free gpu memory: [384] iter= 3490, img_loss=0.028039, reg_loss=0.014467, lr=0.00401, time=329.4 ms, rem=35.74 m free gpu memory: [384] iter= 3500, img_loss=0.043384, reg_loss=0.014465, lr=0.00399, time=329.9 ms, rem=35.74 m free gpu memory: [244] iter= 3510, img_loss=0.040482, reg_loss=0.014467, lr=0.00397, time=329.9 ms, rem=35.68 m free gpu memory: [244] iter= 3520, img_loss=0.038659, reg_loss=0.014451, lr=0.00395, time=331.7 ms, rem=35.82 m free gpu memory: [244] iter= 3530, img_loss=0.037499, reg_loss=0.014452, lr=0.00393, time=331.9 ms, rem=35.79 m free gpu memory: [244] iter= 3540, img_loss=0.033989, reg_loss=0.014451, lr=0.00392, time=333.1 ms, rem=35.86 m free gpu memory: [244] iter= 3550, img_loss=0.049099, reg_loss=0.014458, lr=0.00390, time=331.4 ms, rem=35.63 m free gpu memory: [244] iter= 3560, img_loss=0.051786, reg_loss=0.014463, lr=0.00388, time=330.5 ms, rem=35.47 m free gpu memory: [244] iter= 3570, img_loss=0.049705, reg_loss=0.014464, lr=0.00386, time=330.9 ms, rem=35.46 m free gpu memory: [244] iter= 3580, img_loss=0.067828, reg_loss=0.014451, lr=0.00384, time=331.8 ms, rem=35.50 m free gpu memory: [244] iter= 3590, img_loss=0.020308, reg_loss=0.014460, lr=0.00383, time=330.8 ms, rem=35.34 m free gpu memory: [244] iter= 3600, img_loss=0.032231, reg_loss=0.014470, lr=0.00381, time=331.2 ms, rem=35.33 m free gpu memory: [244] iter= 3610, img_loss=0.072413, reg_loss=0.014455, lr=0.00379, time=331.4 ms, rem=35.29 m free gpu memory: [244] iter= 3620, img_loss=0.065301, reg_loss=0.014455, lr=0.00377, time=331.2 ms, rem=35.22 m free gpu memory: [244] iter= 3630, img_loss=0.060271, reg_loss=0.014458, lr=0.00376, time=330.4 ms, rem=35.08 m free gpu memory: [244] iter= 3640, img_loss=0.024157, reg_loss=0.014459, lr=0.00374, time=330.3 ms, rem=35.01 m free gpu memory: [244] iter= 3650, img_loss=0.053330, reg_loss=0.014449, lr=0.00372, time=332.3 ms, rem=35.17 m free gpu memory: [244] iter= 3660, img_loss=0.043905, reg_loss=0.014458, lr=0.00371, time=331.1 ms, rem=34.99 m free gpu memory: [244] iter= 3670, img_loss=0.059496, reg_loss=0.014449, lr=0.00369, time=331.8 ms, rem=35.01 m free gpu memory: [244] iter= 3680, img_loss=0.069767, reg_loss=0.014456, lr=0.00367, time=331.0 ms, rem=34.87 m free gpu memory: [244] iter= 3690, img_loss=0.056355, reg_loss=0.014458, lr=0.00365, time=329.7 ms, rem=34.67 m free gpu memory: [244] iter= 3700, img_loss=0.040281, reg_loss=0.014460, lr=0.00364, time=331.2 ms, rem=34.78 m free gpu memory: [244] iter= 3710, img_loss=0.072548, reg_loss=0.014459, lr=0.00362, time=331.8 ms, rem=34.78 m free gpu memory: [244] iter= 3720, img_loss=0.060342, reg_loss=0.014463, lr=0.00360, time=330.3 ms, rem=34.57 m free gpu memory: [244] iter= 3730, img_loss=0.035983, reg_loss=0.014471, lr=0.00359, time=328.8 ms, rem=34.36 m free gpu memory: [244] iter= 3740, img_loss=0.042478, reg_loss=0.014468, lr=0.00357, time=330.7 ms, rem=34.50 m free gpu memory: [244] iter= 3750, img_loss=0.059039, reg_loss=0.014464, lr=0.00355, time=329.5 ms, rem=34.32 m free gpu memory: [244] iter= 3760, img_loss=0.032534, reg_loss=0.014471, lr=0.00354, time=330.0 ms, rem=34.32 m free gpu memory: [244] iter= 3770, img_loss=0.030481, reg_loss=0.014448, lr=0.00352, time=330.4 ms, rem=34.31 m free gpu memory: [244] iter= 3780, img_loss=0.085539, reg_loss=0.014439, lr=0.00351, time=333.1 ms, rem=34.53 m free gpu memory: [244] iter= 3790, img_loss=0.020136, reg_loss=0.014462, lr=0.00349, time=330.1 ms, rem=34.17 m free gpu memory: [244] iter= 3800, img_loss=0.091080, reg_loss=0.014451, lr=0.00347, time=331.0 ms, rem=34.20 m free gpu memory: [244] iter= 3810, img_loss=0.072045, reg_loss=0.014457, lr=0.00346, time=331.4 ms, rem=34.19 m free gpu memory: [244] iter= 3820, img_loss=0.023889, reg_loss=0.014460, lr=0.00344, time=329.2 ms, rem=33.91 m free gpu memory: [244] iter= 3830, img_loss=0.035535, reg_loss=0.014466, lr=0.00343, time=329.6 ms, rem=33.89 m free gpu memory: [244] iter= 3840, img_loss=0.048912, reg_loss=0.014457, lr=0.00341, time=331.1 ms, rem=33.99 m free gpu memory: [244] iter= 3850, img_loss=0.071593, reg_loss=0.014452, lr=0.00339, time=332.5 ms, rem=34.08 m free gpu memory: [244] iter= 3860, img_loss=0.073988, reg_loss=0.014455, lr=0.00338, time=331.1 ms, rem=33.88 m free gpu memory: [244] iter= 3870, img_loss=0.027881, reg_loss=0.014459, lr=0.00336, time=330.2 ms, rem=33.74 m free gpu memory: [244] iter= 3880, img_loss=0.018528, reg_loss=0.014470, lr=0.00335, time=328.6 ms, rem=33.52 m free gpu memory: [244] iter= 3890, img_loss=0.029612, reg_loss=0.014460, lr=0.00333, time=330.5 ms, rem=33.66 m free gpu memory: [244] iter= 3900, img_loss=0.070848, reg_loss=0.014455, lr=0.00332, time=331.8 ms, rem=33.73 m free gpu memory: [244] iter= 3910, img_loss=0.024167, reg_loss=0.014480, lr=0.00330, time=340.5 ms, rem=34.56 m free gpu memory: [232] iter= 3920, img_loss=0.046191, reg_loss=0.014462, lr=0.00329, time=342.6 ms, rem=34.72 m free gpu memory: [232] iter= 3930, img_loss=0.053222, reg_loss=0.014467, lr=0.00327, time=340.4 ms, rem=34.44 m free gpu memory: [232] iter= 3940, img_loss=0.020906, reg_loss=0.014482, lr=0.00326, time=339.7 ms, rem=34.31 m free gpu memory: [232] iter= 3950, img_loss=0.052259, reg_loss=0.014459, lr=0.00324, time=342.1 ms, rem=34.50 m free gpu memory: [232] iter= 3960, img_loss=0.035427, reg_loss=0.014466, lr=0.00323, time=341.9 ms, rem=34.42 m free gpu memory: [232] iter= 3970, img_loss=0.011677, reg_loss=0.014473, lr=0.00321, time=338.1 ms, rem=33.98 m free gpu memory: [232] iter= 3980, img_loss=0.089198, reg_loss=0.014450, lr=0.00320, time=342.3 ms, rem=34.35 m free gpu memory: [232] iter= 3990, img_loss=0.056550, reg_loss=0.014459, lr=0.00318, time=341.9 ms, rem=34.25 m free gpu memory: [232] iter= 4000, img_loss=0.062530, reg_loss=0.014448, lr=0.00317, time=344.6 ms, rem=34.46 m free gpu memory: [232] iter= 4010, img_loss=0.042489, reg_loss=0.014451, lr=0.00315, time=343.6 ms, rem=34.30 m free gpu memory: [232] iter= 4020, img_loss=0.029956, reg_loss=0.014464, lr=0.00314, time=340.9 ms, rem=33.98 m free gpu memory: [232] iter= 4030, img_loss=0.083593, reg_loss=0.014454, lr=0.00312, time=341.4 ms, rem=33.97 m free gpu memory: [232] iter= 4040, img_loss=0.063287, reg_loss=0.014455, lr=0.00311, time=343.9 ms, rem=34.16 m free gpu memory: [232] iter= 4050, img_loss=0.053375, reg_loss=0.014449, lr=0.00310, time=342.3 ms, rem=33.95 m free gpu memory: [232] iter= 4060, img_loss=0.031369, reg_loss=0.014462, lr=0.00308, time=339.9 ms, rem=33.65 m free gpu memory: [232] iter= 4070, img_loss=0.030851, reg_loss=0.014466, lr=0.00307, time=340.6 ms, rem=33.66 m free gpu memory: [232] iter= 4080, img_loss=0.035146, reg_loss=0.014454, lr=0.00305, time=340.7 ms, rem=33.62 m free gpu memory: [232] iter= 4090, img_loss=0.051875, reg_loss=0.014458, lr=0.00304, time=340.5 ms, rem=33.54 m free gpu memory: [232] iter= 4100, img_loss=0.054795, reg_loss=0.014445, lr=0.00303, time=344.7 ms, rem=33.90 m free gpu memory: [232] iter= 4110, img_loss=0.013633, reg_loss=0.014461, lr=0.00301, time=341.6 ms, rem=33.53 m free gpu memory: [232] iter= 4120, img_loss=0.068046, reg_loss=0.014456, lr=0.00300, time=340.0 ms, rem=33.32 m free gpu memory: [232] iter= 4130, img_loss=0.052333, reg_loss=0.014444, lr=0.00298, time=340.8 ms, rem=33.34 m free gpu memory: [232] iter= 4140, img_loss=0.092384, reg_loss=0.014429, lr=0.00297, time=342.8 ms, rem=33.48 m free gpu memory: [232] iter= 4150, img_loss=0.062403, reg_loss=0.014451, lr=0.00296, time=339.6 ms, rem=33.11 m free gpu memory: [232] iter= 4160, img_loss=0.039811, reg_loss=0.014459, lr=0.00294, time=338.4 ms, rem=32.94 m free gpu memory: [232] iter= 4170, img_loss=0.020104, reg_loss=0.014453, lr=0.00293, time=341.5 ms, rem=33.18 m free gpu memory: [232] iter= 4180, img_loss=0.050158, reg_loss=0.014443, lr=0.00292, time=341.8 ms, rem=33.16 m free gpu memory: [232] iter= 4190, img_loss=0.052487, reg_loss=0.014446, lr=0.00290, time=341.5 ms, rem=33.07 m free gpu memory: [232] iter= 4200, img_loss=0.029218, reg_loss=0.014453, lr=0.00289, time=340.3 ms, rem=32.90 m free gpu memory: [232] iter= 4210, img_loss=0.024885, reg_loss=0.014470, lr=0.00288, time=340.6 ms, rem=32.87 m free gpu memory: [232] iter= 4220, img_loss=0.027978, reg_loss=0.014459, lr=0.00286, time=340.3 ms, rem=32.78 m free gpu memory: [232] iter= 4230, img_loss=0.034708, reg_loss=0.014457, lr=0.00285, time=341.4 ms, rem=32.83 m free gpu memory: [232] iter= 4240, img_loss=0.074524, reg_loss=0.014454, lr=0.00284, time=340.4 ms, rem=32.68 m free gpu memory: [232] iter= 4250, img_loss=0.037642, reg_loss=0.014447, lr=0.00282, time=341.1 ms, rem=32.69 m free gpu memory: [232] iter= 4260, img_loss=0.037254, reg_loss=0.014437, lr=0.00281, time=341.6 ms, rem=32.68 m free gpu memory: [232] iter= 4270, img_loss=0.040574, reg_loss=0.014447, lr=0.00280, time=340.1 ms, rem=32.48 m free gpu memory: [232] iter= 4280, img_loss=0.059057, reg_loss=0.014438, lr=0.00279, time=341.5 ms, rem=32.56 m free gpu memory: [232] iter= 4290, img_loss=0.061616, reg_loss=0.014428, lr=0.00277, time=342.8 ms, rem=32.62 m free gpu memory: [232] iter= 4300, img_loss=0.039389, reg_loss=0.014452, lr=0.00276, time=340.4 ms, rem=32.34 m free gpu memory: [232] iter= 4310, img_loss=0.052421, reg_loss=0.014449, lr=0.00275, time=342.6 ms, rem=32.49 m free gpu memory: [232] iter= 4320, img_loss=0.027281, reg_loss=0.014452, lr=0.00273, time=341.1 ms, rem=32.29 m free gpu memory: [232] iter= 4330, img_loss=0.041208, reg_loss=0.014460, lr=0.00272, time=340.5 ms, rem=32.18 m free gpu memory: [232] iter= 4340, img_loss=0.051519, reg_loss=0.014442, lr=0.00271, time=341.2 ms, rem=32.19 m free gpu memory: [232] iter= 4350, img_loss=0.052489, reg_loss=0.014449, lr=0.00270, time=341.7 ms, rem=32.18 m free gpu memory: [232] iter= 4360, img_loss=0.078060, reg_loss=0.014424, lr=0.00268, time=344.4 ms, rem=32.37 m free gpu memory: [232] iter= 4370, img_loss=0.054358, reg_loss=0.014434, lr=0.00267, time=343.6 ms, rem=32.24 m free gpu memory: [232] iter= 4380, img_loss=0.041597, reg_loss=0.014445, lr=0.00266, time=338.7 ms, rem=31.73 m free gpu memory: [232] iter= 4390, img_loss=0.030674, reg_loss=0.014443, lr=0.00265, time=343.6 ms, rem=32.13 m free gpu memory: [232] iter= 4400, img_loss=0.043252, reg_loss=0.014451, lr=0.00264, time=341.7 ms, rem=31.89 m free gpu memory: [232] iter= 4410, img_loss=0.056212, reg_loss=0.014438, lr=0.00262, time=348.4 ms, rem=32.46 m free gpu memory: [108] iter= 4420, img_loss=0.058952, reg_loss=0.014442, lr=0.00261, time=342.0 ms, rem=31.81 m free gpu memory: [108] iter= 4430, img_loss=0.044608, reg_loss=0.014450, lr=0.00260, time=339.0 ms, rem=31.47 m free gpu memory: [108] iter= 4440, img_loss=0.023633, reg_loss=0.014450, lr=0.00259, time=339.6 ms, rem=31.47 m free gpu memory: [108] iter= 4450, img_loss=0.077172, reg_loss=0.014435, lr=0.00258, time=341.2 ms, rem=31.56 m free gpu memory: [108] iter= 4460, img_loss=0.062358, reg_loss=0.014444, lr=0.00256, time=340.3 ms, rem=31.42 m free gpu memory: [108] iter= 4470, img_loss=0.070660, reg_loss=0.014434, lr=0.00255, time=342.2 ms, rem=31.54 m free gpu memory: [108] iter= 4480, img_loss=0.026964, reg_loss=0.014448, lr=0.00254, time=338.9 ms, rem=31.18 m free gpu memory: [108] iter= 4490, img_loss=0.041796, reg_loss=0.014442, lr=0.00253, time=342.1 ms, rem=31.42 m free gpu memory: [108] iter= 4500, img_loss=0.044553, reg_loss=0.014443, lr=0.00252, time=342.5 ms, rem=31.40 m free gpu memory: [108] iter= 4510, img_loss=0.063403, reg_loss=0.014428, lr=0.00251, time=345.5 ms, rem=31.61 m free gpu memory: [108] iter= 4520, img_loss=0.065251, reg_loss=0.014430, lr=0.00249, time=341.2 ms, rem=31.16 m free gpu memory: [108] iter= 4530, img_loss=0.040791, reg_loss=0.014445, lr=0.00248, time=342.1 ms, rem=31.19 m free gpu memory: [108] iter= 4540, img_loss=0.038549, reg_loss=0.014449, lr=0.00247, time=340.2 ms, rem=30.96 m free gpu memory: [108] iter= 4550, img_loss=0.026643, reg_loss=0.014453, lr=0.00246, time=339.8 ms, rem=30.87 m free gpu memory: [108] iter= 4560, img_loss=0.066395, reg_loss=0.014441, lr=0.00245, time=342.0 ms, rem=31.01 m free gpu memory: [108] iter= 4570, img_loss=0.048082, reg_loss=0.014440, lr=0.00244, time=341.7 ms, rem=30.92 m free gpu memory: [108] iter= 4580, img_loss=0.033361, reg_loss=0.014439, lr=0.00243, time=341.0 ms, rem=30.80 m free gpu memory: [108] iter= 4590, img_loss=0.084913, reg_loss=0.014415, lr=0.00241, time=345.5 ms, rem=31.15 m free gpu memory: [108] iter= 4600, img_loss=0.078979, reg_loss=0.014430, lr=0.00240, time=341.5 ms, rem=30.74 m free gpu memory: [108] iter= 4610, img_loss=0.068293, reg_loss=0.014431, lr=0.00239, time=342.4 ms, rem=30.76 m free gpu memory: [108] iter= 4620, img_loss=0.030850, reg_loss=0.014435, lr=0.00238, time=341.5 ms, rem=30.62 m free gpu memory: [108] iter= 4630, img_loss=0.066655, reg_loss=0.014427, lr=0.00237, time=343.0 ms, rem=30.70 m free gpu memory: [108] iter= 4640, img_loss=0.066549, reg_loss=0.014430, lr=0.00236, time=343.0 ms, rem=30.64 m free gpu memory: [108] iter= 4650, img_loss=0.045452, reg_loss=0.014442, lr=0.00235, time=339.3 ms, rem=30.26 m free gpu memory: [108] iter= 4660, img_loss=0.029421, reg_loss=0.014437, lr=0.00234, time=341.0 ms, rem=30.35 m free gpu memory: [108] iter= 4670, img_loss=0.058386, reg_loss=0.014429, lr=0.00233, time=343.2 ms, rem=30.49 m free gpu memory: [108] iter= 4680, img_loss=0.021564, reg_loss=0.014459, lr=0.00232, time=339.0 ms, rem=30.06 m free gpu memory: [108] iter= 4690, img_loss=0.025411, reg_loss=0.014451, lr=0.00231, time=341.3 ms, rem=30.21 m free gpu memory: [108] iter= 4700, img_loss=0.059513, reg_loss=0.014437, lr=0.00230, time=342.8 ms, rem=30.28 m free gpu memory: [108] iter= 4710, img_loss=0.048999, reg_loss=0.014441, lr=0.00228, time=345.3 ms, rem=30.45 m free gpu memory: [108] iter= 4720, img_loss=0.059370, reg_loss=0.014442, lr=0.00227, time=340.1 ms, rem=29.93 m free gpu memory: [108] iter= 4730, img_loss=0.034117, reg_loss=0.014437, lr=0.00226, time=341.3 ms, rem=29.98 m free gpu memory: [108] iter= 4740, img_loss=0.059594, reg_loss=0.014449, lr=0.00225, time=339.8 ms, rem=29.79 m free gpu memory: [108] iter= 4750, img_loss=0.056621, reg_loss=0.014437, lr=0.00224, time=339.5 ms, rem=29.71 m free gpu memory: [108] iter= 4760, img_loss=0.063040, reg_loss=0.014438, lr=0.00223, time=340.5 ms, rem=29.74 m free gpu memory: [108] iter= 4770, img_loss=0.015426, reg_loss=0.014443, lr=0.00222, time=339.2 ms, rem=29.57 m free gpu memory: [108] iter= 4780, img_loss=0.035337, reg_loss=0.014433, lr=0.00221, time=341.8 ms, rem=29.74 m free gpu memory: [108] iter= 4790, img_loss=0.025151, reg_loss=0.014440, lr=0.00220, time=339.7 ms, rem=29.50 m free gpu memory: [108] iter= 4800, img_loss=0.095866, reg_loss=0.014428, lr=0.00219, time=346.2 ms, rem=30.00 m free gpu memory: [108] iter= 4810, img_loss=0.042642, reg_loss=0.014449, lr=0.00218, time=341.7 ms, rem=29.56 m free gpu memory: [108] iter= 4820, img_loss=0.040236, reg_loss=0.014436, lr=0.00217, time=342.9 ms, rem=29.60 m free gpu memory: [108] iter= 4830, img_loss=0.082435, reg_loss=0.014424, lr=0.00216, time=343.7 ms, rem=29.62 m free gpu memory: [108] iter= 4840, img_loss=0.054195, reg_loss=0.014425, lr=0.00215, time=342.3 ms, rem=29.44 m free gpu memory: [108] iter= 4850, img_loss=0.043490, reg_loss=0.014439, lr=0.00214, time=340.6 ms, rem=29.24 m free gpu memory: [108] iter= 4860, img_loss=0.073999, reg_loss=0.014439, lr=0.00213, time=342.2 ms, rem=29.32 m free gpu memory: [108] iter= 4870, img_loss=0.045307, reg_loss=0.014436, lr=0.00212, time=342.8 ms, rem=29.31 m free gpu memory: [108] iter= 4880, img_loss=0.019134, reg_loss=0.014442, lr=0.00211, time=340.1 ms, rem=29.02 m free gpu memory: [108] iter= 4890, img_loss=0.041182, reg_loss=0.014434, lr=0.00210, time=341.7 ms, rem=29.10 m free gpu memory: [108] iter= 4900, img_loss=0.045946, reg_loss=0.014434, lr=0.00209, time=342.8 ms, rem=29.14 m free gpu memory: [108] iter= 4910, img_loss=0.037663, reg_loss=0.014433, lr=0.00208, time=344.1 ms, rem=29.19 m free gpu memory: [108] iter= 4920, img_loss=0.029235, reg_loss=0.014442, lr=0.00207, time=338.9 ms, rem=28.69 m free gpu memory: [108] iter= 4930, img_loss=0.023541, reg_loss=0.014438, lr=0.00206, time=340.9 ms, rem=28.81 m free gpu memory: [108] iter= 4940, img_loss=0.030850, reg_loss=0.014435, lr=0.00206, time=343.3 ms, rem=28.95 m free gpu memory: [108] iter= 4950, img_loss=0.041696, reg_loss=0.014430, lr=0.00205, time=340.2 ms, rem=28.63 m free gpu memory: [108] iter= 4960, img_loss=0.058570, reg_loss=0.014432, lr=0.00204, time=342.1 ms, rem=28.74 m free gpu memory: [108] iter= 4970, img_loss=0.053950, reg_loss=0.014432, lr=0.00203, time=341.4 ms, rem=28.62 m free gpu memory: [108] iter= 4980, img_loss=0.036673, reg_loss=0.014432, lr=0.00202, time=342.1 ms, rem=28.62 m free gpu memory: [108] iter= 4990, img_loss=0.041716, reg_loss=0.014423, lr=0.00201, time=344.4 ms, rem=28.76 m free gpu memory: [108] iter= 5000, img_loss=0.040680, reg_loss=0.014443, lr=0.00200, time=339.0 ms, rem=28.25 m free gpu memory: [108] iter= 5010, img_loss=0.106084, reg_loss=0.014416, lr=0.00199, time=345.5 ms, rem=28.74 m free gpu memory: [108] iter= 5020, img_loss=0.035635, reg_loss=0.014422, lr=0.00198, time=341.7 ms, rem=28.36 m free gpu memory: [108] iter= 5030, img_loss=0.070307, reg_loss=0.014424, lr=0.00197, time=341.2 ms, rem=28.26 m free gpu memory: [108] iter= 5040, img_loss=0.031565, reg_loss=0.014432, lr=0.00196, time=341.9 ms, rem=28.26 m free gpu memory: [108] iter= 5050, img_loss=0.072286, reg_loss=0.014419, lr=0.00195, time=339.6 ms, rem=28.02 m free gpu memory: [108] iter= 5060, img_loss=0.044205, reg_loss=0.014437, lr=0.00194, time=340.6 ms, rem=28.04 m free gpu memory: [108] iter= 5070, img_loss=0.032142, reg_loss=0.014439, lr=0.00194, time=337.8 ms, rem=27.76 m free gpu memory: [108] iter= 5080, img_loss=0.041257, reg_loss=0.014435, lr=0.00193, time=341.7 ms, rem=28.02 m free gpu memory: [108] iter= 5090, img_loss=0.060250, reg_loss=0.014417, lr=0.00192, time=342.9 ms, rem=28.06 m free gpu memory: [108] iter= 5100, img_loss=0.067819, reg_loss=0.014424, lr=0.00191, time=342.7 ms, rem=27.99 m free gpu memory: [108] iter= 5110, img_loss=0.020413, reg_loss=0.014443, lr=0.00190, time=341.4 ms, rem=27.82 m free gpu memory: [108] iter= 5120, img_loss=0.055703, reg_loss=0.014419, lr=0.00189, time=342.4 ms, rem=27.85 m free gpu memory: [108] iter= 5130, img_loss=0.061646, reg_loss=0.014433, lr=0.00188, time=340.5 ms, rem=27.64 m free gpu memory: [108] iter= 5140, img_loss=0.016596, reg_loss=0.014439, lr=0.00187, time=339.1 ms, rem=27.47 m free gpu memory: [108] iter= 5150, img_loss=0.055370, reg_loss=0.014414, lr=0.00187, time=341.9 ms, rem=27.64 m free gpu memory: [108] iter= 5160, img_loss=0.048384, reg_loss=0.014428, lr=0.00186, time=341.3 ms, rem=27.53 m free gpu memory: [108] iter= 5170, img_loss=0.011501, reg_loss=0.014427, lr=0.00185, time=339.8 ms, rem=27.35 m free gpu memory: [108] iter= 5180, img_loss=0.087151, reg_loss=0.014417, lr=0.00184, time=344.0 ms, rem=27.64 m free gpu memory: [108] iter= 5190, img_loss=0.054587, reg_loss=0.014410, lr=0.00183, time=342.5 ms, rem=27.46 m free gpu memory: [108] iter= 5200, img_loss=0.065858, reg_loss=0.014420, lr=0.00182, time=342.6 ms, rem=27.41 m free gpu memory: [108] iter= 5210, img_loss=0.071512, reg_loss=0.014417, lr=0.00181, time=350.3 ms, rem=27.97 m free gpu memory: [108] iter= 5220, img_loss=0.041073, reg_loss=0.014422, lr=0.00181, time=343.2 ms, rem=27.34 m free gpu memory: [109] iter= 5230, img_loss=0.048869, reg_loss=0.014435, lr=0.00180, time=339.9 ms, rem=27.02 m free gpu memory: [109] iter= 5240, img_loss=0.029286, reg_loss=0.014422, lr=0.00179, time=343.1 ms, rem=27.22 m free gpu memory: [109] iter= 5250, img_loss=0.020874, reg_loss=0.014431, lr=0.00178, time=340.2 ms, rem=26.93 m free gpu memory: [109] iter= 5260, img_loss=0.096964, reg_loss=0.014406, lr=0.00177, time=346.4 ms, rem=27.37 m free gpu memory: [109] iter= 5270, img_loss=0.031118, reg_loss=0.014430, lr=0.00177, time=341.5 ms, rem=26.92 m free gpu memory: [109] iter= 5280, img_loss=0.037515, reg_loss=0.014422, lr=0.00176, time=342.8 ms, rem=26.97 m free gpu memory: [109] iter= 5290, img_loss=0.048143, reg_loss=0.014417, lr=0.00175, time=344.1 ms, rem=27.01 m free gpu memory: [109] iter= 5300, img_loss=0.052862, reg_loss=0.014421, lr=0.00174, time=343.7 ms, rem=26.92 m free gpu memory: [109] iter= 5310, img_loss=0.071916, reg_loss=0.014408, lr=0.00173, time=346.5 ms, rem=27.09 m free gpu memory: [109] iter= 5320, img_loss=0.056720, reg_loss=0.014423, lr=0.00173, time=339.8 ms, rem=26.51 m free gpu memory: [109] iter= 5330, img_loss=0.026428, reg_loss=0.014423, lr=0.00172, time=340.0 ms, rem=26.46 m free gpu memory: [109] iter= 5340, img_loss=0.034332, reg_loss=0.014429, lr=0.00171, time=341.5 ms, rem=26.52 m free gpu memory: [109] iter= 5350, img_loss=0.011053, reg_loss=0.014440, lr=0.00170, time=336.6 ms, rem=26.09 m free gpu memory: [109] iter= 5360, img_loss=0.023547, reg_loss=0.014428, lr=0.00169, time=339.1 ms, rem=26.22 m free gpu memory: [109] iter= 5370, img_loss=0.051341, reg_loss=0.014407, lr=0.00169, time=341.4 ms, rem=26.35 m free gpu memory: [109] iter= 5380, img_loss=0.058958, reg_loss=0.014424, lr=0.00168, time=343.5 ms, rem=26.45 m free gpu memory: [109] iter= 5390, img_loss=0.071688, reg_loss=0.014423, lr=0.00167, time=337.6 ms, rem=25.94 m free gpu memory: [109] iter= 5400, img_loss=0.029558, reg_loss=0.014428, lr=0.00166, time=341.5 ms, rem=26.18 m free gpu memory: [109] iter= 5410, img_loss=0.026418, reg_loss=0.014426, lr=0.00166, time=340.2 ms, rem=26.03 m free gpu memory: [109] iter= 5420, img_loss=0.039787, reg_loss=0.014415, lr=0.00165, time=341.7 ms, rem=26.08 m free gpu memory: [109] iter= 5430, img_loss=0.037050, reg_loss=0.014419, lr=0.00164, time=340.7 ms, rem=25.95 m free gpu memory: [109] iter= 5440, img_loss=0.043319, reg_loss=0.014418, lr=0.00163, time=342.5 ms, rem=26.03 m free gpu memory: [109] iter= 5450, img_loss=0.051560, reg_loss=0.014414, lr=0.00162, time=340.7 ms, rem=25.84 m free gpu memory: [109] iter= 5460, img_loss=0.058833, reg_loss=0.014412, lr=0.00162, time=342.0 ms, rem=25.88 m free gpu memory: [109] iter= 5470, img_loss=0.046808, reg_loss=0.014416, lr=0.00161, time=340.1 ms, rem=25.68 m free gpu memory: [109] iter= 5480, img_loss=0.021052, reg_loss=0.014425, lr=0.00160, time=341.1 ms, rem=25.70 m free gpu memory: [109] iter= 5490, img_loss=0.066651, reg_loss=0.014415, lr=0.00160, time=339.9 ms, rem=25.55 m free gpu memory: [109] iter= 5500, img_loss=0.028996, reg_loss=0.014426, lr=0.00159, time=341.8 ms, rem=25.64 m free gpu memory: [109] iter= 5510, img_loss=0.047079, reg_loss=0.014413, lr=0.00158, time=343.5 ms, rem=25.71 m free gpu memory: [109] iter= 5520, img_loss=0.043845, reg_loss=0.014414, lr=0.00157, time=342.0 ms, rem=25.54 m free gpu memory: [109] iter= 5530, img_loss=0.018812, reg_loss=0.014420, lr=0.00157, time=341.3 ms, rem=25.43 m free gpu memory: [109] iter= 5540, img_loss=0.021449, reg_loss=0.014408, lr=0.00156, time=342.1 ms, rem=25.43 m free gpu memory: [109] iter= 5550, img_loss=0.045990, reg_loss=0.014416, lr=0.00155, time=342.1 ms, rem=25.37 m free gpu memory: [109] iter= 5560, img_loss=0.070162, reg_loss=0.014405, lr=0.00154, time=342.3 ms, rem=25.33 m free gpu memory: [109] iter= 5570, img_loss=0.062798, reg_loss=0.014400, lr=0.00154, time=341.5 ms, rem=25.21 m free gpu memory: [109] iter= 5580, img_loss=0.073065, reg_loss=0.014403, lr=0.00153, time=343.9 ms, rem=25.33 m free gpu memory: [109] iter= 5590, img_loss=0.046712, reg_loss=0.014420, lr=0.00152, time=337.6 ms, rem=24.81 m free gpu memory: [109] iter= 5600, img_loss=0.047142, reg_loss=0.014417, lr=0.00152, time=347.0 ms, rem=25.45 m free gpu memory: [100] iter= 5610, img_loss=0.020146, reg_loss=0.014416, lr=0.00151, time=344.6 ms, rem=25.21 m free gpu memory: [101] iter= 5620, img_loss=0.071123, reg_loss=0.014405, lr=0.00150, time=343.0 ms, rem=25.04 m free gpu memory: [101] iter= 5630, img_loss=0.050918, reg_loss=0.014412, lr=0.00150, time=343.1 ms, rem=24.99 m free gpu memory: [101] iter= 5640, img_loss=0.050392, reg_loss=0.014404, lr=0.00149, time=341.8 ms, rem=24.84 m free gpu memory: [101] iter= 5650, img_loss=0.030987, reg_loss=0.014415, lr=0.00148, time=341.3 ms, rem=24.75 m free gpu memory: [90] iter= 5660, img_loss=0.050296, reg_loss=0.014409, lr=0.00148, time=342.1 ms, rem=24.75 m free gpu memory: [90] iter= 5670, img_loss=0.061899, reg_loss=0.014408, lr=0.00147, time=343.1 ms, rem=24.76 m free gpu memory: [89] iter= 5680, img_loss=0.080786, reg_loss=0.014396, lr=0.00146, time=344.6 ms, rem=24.81 m free gpu memory: [89] iter= 5690, img_loss=0.053919, reg_loss=0.014394, lr=0.00145, time=343.6 ms, rem=24.68 m free gpu memory: [90] iter= 5700, img_loss=0.059133, reg_loss=0.014407, lr=0.00145, time=340.7 ms, rem=24.42 m free gpu memory: [90] iter= 5710, img_loss=0.055667, reg_loss=0.014410, lr=0.00144, time=342.4 ms, rem=24.48 m free gpu memory: [90] iter= 5720, img_loss=0.042506, reg_loss=0.014409, lr=0.00143, time=340.6 ms, rem=24.30 m free gpu memory: [90] iter= 5730, img_loss=0.082963, reg_loss=0.014401, lr=0.00143, time=343.6 ms, rem=24.45 m free gpu memory: [90] iter= 5740, img_loss=0.092652, reg_loss=0.014388, lr=0.00142, time=344.4 ms, rem=24.45 m free gpu memory: [90] iter= 5750, img_loss=0.046381, reg_loss=0.014389, lr=0.00142, time=344.4 ms, rem=24.40 m free gpu memory: [90] iter= 5760, img_loss=0.075777, reg_loss=0.014397, lr=0.00141, time=342.5 ms, rem=24.20 m free gpu memory: [90] iter= 5770, img_loss=0.053220, reg_loss=0.014402, lr=0.00140, time=342.7 ms, rem=24.16 m free gpu memory: [90] iter= 5780, img_loss=0.064195, reg_loss=0.014398, lr=0.00140, time=342.0 ms, rem=24.05 m free gpu memory: [90] iter= 5790, img_loss=0.038947, reg_loss=0.014410, lr=0.00139, time=340.2 ms, rem=23.87 m free gpu memory: [90] iter= 5800, img_loss=0.011490, reg_loss=0.014408, lr=0.00138, time=341.7 ms, rem=23.92 m free gpu memory: [89] iter= 5810, img_loss=0.040650, reg_loss=0.014405, lr=0.00138, time=344.1 ms, rem=24.03 m free gpu memory: [90] iter= 5820, img_loss=0.058454, reg_loss=0.014398, lr=0.00137, time=340.5 ms, rem=23.72 m free gpu memory: [90] iter= 5830, img_loss=0.026749, reg_loss=0.014399, lr=0.00136, time=339.8 ms, rem=23.62 m free gpu memory: [90] iter= 5840, img_loss=0.066004, reg_loss=0.014382, lr=0.00136, time=343.8 ms, rem=23.84 m free gpu memory: [90] iter= 5850, img_loss=0.050845, reg_loss=0.014396, lr=0.00135, time=341.6 ms, rem=23.63 m free gpu memory: [90] iter= 5860, img_loss=0.069538, reg_loss=0.014389, lr=0.00135, time=342.2 ms, rem=23.61 m free gpu memory: [90] iter= 5870, img_loss=0.041450, reg_loss=0.014405, lr=0.00134, time=341.3 ms, rem=23.49 m free gpu memory: [90] iter= 5880, img_loss=0.050952, reg_loss=0.014396, lr=0.00133, time=339.8 ms, rem=23.33 m free gpu memory: [90] iter= 5890, img_loss=0.073667, reg_loss=0.014399, lr=0.00133, time=340.1 ms, rem=23.30 m free gpu memory: [90] iter= 5900, img_loss=0.070368, reg_loss=0.014396, lr=0.00132, time=341.3 ms, rem=23.32 m free gpu memory: [90] iter= 5910, img_loss=0.019872, reg_loss=0.014405, lr=0.00131, time=369.0 ms, rem=25.16 m free gpu memory: [78] iter= 5920, img_loss=0.060982, reg_loss=0.014396, lr=0.00131, time=345.0 ms, rem=23.46 m free gpu memory: [78] iter= 5930, img_loss=0.047298, reg_loss=0.014394, lr=0.00130, time=344.6 ms, rem=23.38 m free gpu memory: [78] iter= 5940, img_loss=0.023790, reg_loss=0.014408, lr=0.00130, time=345.0 ms, rem=23.35 m free gpu memory: [78] iter= 5950, img_loss=0.018029, reg_loss=0.014386, lr=0.00129, time=344.8 ms, rem=23.27 m free gpu memory: [89] iter= 5960, img_loss=0.084282, reg_loss=0.014376, lr=0.00128, time=348.9 ms, rem=23.49 m free gpu memory: [89] iter= 5970, img_loss=0.022028, reg_loss=0.014412, lr=0.00128, time=343.4 ms, rem=23.07 m free gpu memory: [89] iter= 5980, img_loss=0.037282, reg_loss=0.014390, lr=0.00127, time=347.4 ms, rem=23.28 m free gpu memory: [89] iter= 5990, img_loss=0.060128, reg_loss=0.014377, lr=0.00127, time=346.7 ms, rem=23.17 m free gpu memory: [78] iter= 6000, img_loss=0.032578, reg_loss=0.014396, lr=0.00126, time=345.4 ms, rem=23.03 m free gpu memory: [78] iter= 6010, img_loss=0.036563, reg_loss=0.014399, lr=0.00126, time=346.0 ms, rem=23.01 m free gpu memory: [80] iter= 6020, img_loss=0.081899, reg_loss=0.014373, lr=0.00125, time=346.5 ms, rem=22.99 m free gpu memory: [80] iter= 6030, img_loss=0.034494, reg_loss=0.014389, lr=0.00124, time=345.7 ms, rem=22.87 m free gpu memory: [80] iter= 6040, img_loss=0.032299, reg_loss=0.014387, lr=0.00124, time=345.2 ms, rem=22.78 m free gpu memory: [80] iter= 6050, img_loss=0.032914, reg_loss=0.014384, lr=0.00123, time=346.2 ms, rem=22.79 m free gpu memory: [80] iter= 6060, img_loss=0.044458, reg_loss=0.014391, lr=0.00123, time=343.0 ms, rem=22.52 m free gpu memory: [80] iter= 6070, img_loss=0.037359, reg_loss=0.014383, lr=0.00122, time=343.8 ms, rem=22.52 m free gpu memory: [80] iter= 6080, img_loss=0.036133, reg_loss=0.014398, lr=0.00122, time=344.4 ms, rem=22.50 m free gpu memory: [80] iter= 6090, img_loss=0.032067, reg_loss=0.014387, lr=0.00121, time=345.4 ms, rem=22.51 m free gpu memory: [80] iter= 6100, img_loss=0.011581, reg_loss=0.014394, lr=0.00120, time=342.8 ms, rem=22.28 m free gpu memory: [80] iter= 6110, img_loss=0.039351, reg_loss=0.014383, lr=0.00120, time=346.9 ms, rem=22.49 m free gpu memory: [80] iter= 6120, img_loss=0.044916, reg_loss=0.014380, lr=0.00119, time=345.2 ms, rem=22.32 m free gpu memory: [80] iter= 6130, img_loss=0.041438, reg_loss=0.014379, lr=0.00119, time=347.6 ms, rem=22.42 m free gpu memory: [80] iter= 6140, img_loss=0.050981, reg_loss=0.014389, lr=0.00118, time=344.5 ms, rem=22.16 m free gpu memory: [80] iter= 6150, img_loss=0.021118, reg_loss=0.014398, lr=0.00118, time=344.2 ms, rem=22.09 m free gpu memory: [80] iter= 6160, img_loss=0.025651, reg_loss=0.014391, lr=0.00117, time=344.0 ms, rem=22.02 m free gpu memory: [80] iter= 6170, img_loss=0.083908, reg_loss=0.014372, lr=0.00117, time=346.1 ms, rem=22.09 m free gpu memory: [80] iter= 6180, img_loss=0.037368, reg_loss=0.014388, lr=0.00116, time=346.4 ms, rem=22.06 m free gpu memory: [80] iter= 6190, img_loss=0.026959, reg_loss=0.014384, lr=0.00116, time=344.6 ms, rem=21.88 m free gpu memory: [80] iter= 6200, img_loss=0.041333, reg_loss=0.014389, lr=0.00115, time=343.4 ms, rem=21.75 m free gpu memory: [80] iter= 6210, img_loss=0.041999, reg_loss=0.014375, lr=0.00115, time=349.3 ms, rem=22.07 m free gpu memory: [80] iter= 6220, img_loss=0.068144, reg_loss=0.014374, lr=0.00114, time=346.0 ms, rem=21.80 m free gpu memory: [80] iter= 6230, img_loss=0.052529, reg_loss=0.014380, lr=0.00113, time=344.9 ms, rem=21.67 m free gpu memory: [79] iter= 6240, img_loss=0.073061, reg_loss=0.014373, lr=0.00113, time=343.8 ms, rem=21.55 m free gpu memory: [79] iter= 6250, img_loss=0.074334, reg_loss=0.014382, lr=0.00112, time=345.4 ms, rem=21.59 m free gpu memory: [79] iter= 6260, img_loss=0.064400, reg_loss=0.014367, lr=0.00112, time=346.2 ms, rem=21.58 m free gpu memory: [79] iter= 6270, img_loss=0.052824, reg_loss=0.014380, lr=0.00111, time=344.8 ms, rem=21.44 m free gpu memory: [79] iter= 6280, img_loss=0.025057, reg_loss=0.014395, lr=0.00111, time=343.2 ms, rem=21.28 m free gpu memory: [79] iter= 6290, img_loss=0.030386, reg_loss=0.014390, lr=0.00110, time=344.2 ms, rem=21.28 m free gpu memory: [79] iter= 6300, img_loss=0.069963, reg_loss=0.014371, lr=0.00110, time=348.8 ms, rem=21.51 m free gpu memory: [79] iter= 6310, img_loss=0.023606, reg_loss=0.014387, lr=0.00109, time=346.2 ms, rem=21.29 m free gpu memory: [79] iter= 6320, img_loss=0.025051, reg_loss=0.014387, lr=0.00109, time=345.5 ms, rem=21.19 m free gpu memory: [79] iter= 6330, img_loss=0.041922, reg_loss=0.014383, lr=0.00108, time=344.4 ms, rem=21.07 m free gpu memory: [79] iter= 6340, img_loss=0.073853, reg_loss=0.014363, lr=0.00108, time=349.3 ms, rem=21.31 m free gpu memory: [79] iter= 6350, img_loss=0.047375, reg_loss=0.014382, lr=0.00107, time=343.8 ms, rem=20.92 m free gpu memory: [79] iter= 6360, img_loss=0.026041, reg_loss=0.014380, lr=0.00107, time=346.6 ms, rem=21.03 m free gpu memory: [79] iter= 6370, img_loss=0.044977, reg_loss=0.014381, lr=0.00106, time=344.8 ms, rem=20.86 m free gpu memory: [79] iter= 6380, img_loss=0.052914, reg_loss=0.014378, lr=0.00106, time=343.5 ms, rem=20.73 m free gpu memory: [79] iter= 6390, img_loss=0.059780, reg_loss=0.014369, lr=0.00105, time=345.7 ms, rem=20.80 m free gpu memory: [79] iter= 6400, img_loss=0.038491, reg_loss=0.014371, lr=0.00105, time=348.0 ms, rem=20.88 m free gpu memory: [79] iter= 6410, img_loss=0.031508, reg_loss=0.014368, lr=0.00104, time=382.9 ms, rem=22.91 m free gpu memory: [119] iter= 6420, img_loss=0.040698, reg_loss=0.014380, lr=0.00104, time=341.4 ms, rem=20.37 m free gpu memory: [117] iter= 6430, img_loss=0.036078, reg_loss=0.014373, lr=0.00103, time=340.6 ms, rem=20.27 m free gpu memory: [117] iter= 6440, img_loss=0.019504, reg_loss=0.014375, lr=0.00103, time=341.1 ms, rem=20.24 m free gpu memory: [117] iter= 6450, img_loss=0.054316, reg_loss=0.014363, lr=0.00103, time=344.8 ms, rem=20.40 m free gpu memory: [117] iter= 6460, img_loss=0.041968, reg_loss=0.014366, lr=0.00102, time=342.4 ms, rem=20.20 m free gpu memory: [117] iter= 6470, img_loss=0.067450, reg_loss=0.014364, lr=0.00102, time=346.4 ms, rem=20.38 m free gpu memory: [115] iter= 6480, img_loss=0.036417, reg_loss=0.014367, lr=0.00101, time=341.7 ms, rem=20.05 m free gpu memory: [115] iter= 6490, img_loss=0.046020, reg_loss=0.014364, lr=0.00101, time=340.3 ms, rem=19.91 m free gpu memory: [115] iter= 6500, img_loss=0.026774, reg_loss=0.014367, lr=0.00100, time=340.6 ms, rem=19.87 m free gpu memory: [115] iter= 6510, img_loss=0.047172, reg_loss=0.014361, lr=0.00100, time=346.0 ms, rem=20.13 m free gpu memory: [115] iter= 6520, img_loss=0.056601, reg_loss=0.014359, lr=0.00099, time=342.8 ms, rem=19.88 m free gpu memory: [115] iter= 6530, img_loss=0.036056, reg_loss=0.014371, lr=0.00099, time=344.2 ms, rem=19.91 m free gpu memory: [115] iter= 6540, img_loss=0.022901, reg_loss=0.014377, lr=0.00098, time=341.1 ms, rem=19.67 m free gpu memory: [115] iter= 6550, img_loss=0.040719, reg_loss=0.014361, lr=0.00098, time=343.7 ms, rem=19.76 m free gpu memory: [115] iter= 6560, img_loss=0.047966, reg_loss=0.014354, lr=0.00097, time=344.6 ms, rem=19.76 m free gpu memory: [115] iter= 6570, img_loss=0.030150, reg_loss=0.014369, lr=0.00097, time=342.4 ms, rem=19.57 m free gpu memory: [115] iter= 6580, img_loss=0.062956, reg_loss=0.014358, lr=0.00097, time=345.6 ms, rem=19.70 m free gpu memory: [115] iter= 6590, img_loss=0.046686, reg_loss=0.014367, lr=0.00096, time=362.4 ms, rem=20.60 m free gpu memory: [60] iter= 6600, img_loss=0.057534, reg_loss=0.014358, lr=0.00096, time=343.0 ms, rem=19.44 m free gpu memory: [60] iter= 6610, img_loss=0.035667, reg_loss=0.014355, lr=0.00095, time=344.8 ms, rem=19.48 m free gpu memory: [60] iter= 6620, img_loss=0.051679, reg_loss=0.014355, lr=0.00095, time=344.8 ms, rem=19.42 m free gpu memory: [60] iter= 6630, img_loss=0.025779, reg_loss=0.014350, lr=0.00094, time=346.6 ms, rem=19.47 m free gpu memory: [60] iter= 6640, img_loss=0.047869, reg_loss=0.014364, lr=0.00094, time=343.6 ms, rem=19.24 m free gpu memory: [60] iter= 6650, img_loss=0.032253, reg_loss=0.014369, lr=0.00094, time=341.1 ms, rem=19.05 m free gpu memory: [60] iter= 6660, img_loss=0.018490, reg_loss=0.014363, lr=0.00093, time=342.0 ms, rem=19.04 m free gpu memory: [60] iter= 6670, img_loss=0.074559, reg_loss=0.014354, lr=0.00093, time=342.7 ms, rem=19.02 m free gpu memory: [60] iter= 6680, img_loss=0.019210, reg_loss=0.014369, lr=0.00092, time=342.0 ms, rem=18.92 m free gpu memory: [60] iter= 6690, img_loss=0.051515, reg_loss=0.014360, lr=0.00092, time=341.9 ms, rem=18.86 m free gpu memory: [60] iter= 6700, img_loss=0.037222, reg_loss=0.014358, lr=0.00091, time=342.5 ms, rem=18.84 m free gpu memory: [60] iter= 6710, img_loss=0.067340, reg_loss=0.014361, lr=0.00091, time=343.5 ms, rem=18.84 m free gpu memory: [60] iter= 6720, img_loss=0.039614, reg_loss=0.014357, lr=0.00091, time=342.6 ms, rem=18.73 m free gpu memory: [60] iter= 6730, img_loss=0.039003, reg_loss=0.014355, lr=0.00090, time=341.9 ms, rem=18.63 m free gpu memory: [60] iter= 6740, img_loss=0.040698, reg_loss=0.014356, lr=0.00090, time=341.8 ms, rem=18.57 m free gpu memory: [60] iter= 6750, img_loss=0.032183, reg_loss=0.014361, lr=0.00089, time=339.2 ms, rem=18.37 m free gpu memory: [60] iter= 6760, img_loss=0.038388, reg_loss=0.014353, lr=0.00089, time=341.9 ms, rem=18.46 m free gpu memory: [60] iter= 6770, img_loss=0.023366, reg_loss=0.014360, lr=0.00088, time=342.2 ms, rem=18.42 m free gpu memory: [60] iter= 6780, img_loss=0.015784, reg_loss=0.014368, lr=0.00088, time=339.9 ms, rem=18.24 m free gpu memory: [60] iter= 6790, img_loss=0.038006, reg_loss=0.014349, lr=0.00088, time=341.9 ms, rem=18.29 m free gpu memory: [60] iter= 6800, img_loss=0.061564, reg_loss=0.014347, lr=0.00087, time=346.5 ms, rem=18.48 m free gpu memory: [12] iter= 6810, img_loss=0.030383, reg_loss=0.014360, lr=0.00087, time=341.8 ms, rem=18.17 m free gpu memory: [12] iter= 6820, img_loss=0.054284, reg_loss=0.014353, lr=0.00086, time=342.5 ms, rem=18.15 m free gpu memory: [12] iter= 6830, img_loss=0.061039, reg_loss=0.014351, lr=0.00086, time=340.2 ms, rem=17.97 m free gpu memory: [12] iter= 6840, img_loss=0.039763, reg_loss=0.014355, lr=0.00086, time=341.6 ms, rem=17.99 m free gpu memory: [12] iter= 6850, img_loss=0.100372, reg_loss=0.014340, lr=0.00085, time=343.4 ms, rem=18.03 m free gpu memory: [12] iter= 6860, img_loss=0.022313, reg_loss=0.014359, lr=0.00085, time=340.8 ms, rem=17.84 m free gpu memory: [12] iter= 6870, img_loss=0.025590, reg_loss=0.014357, lr=0.00084, time=342.7 ms, rem=17.88 m free gpu memory: [12] iter= 6880, img_loss=0.036397, reg_loss=0.014363, lr=0.00084, time=339.9 ms, rem=17.68 m free gpu memory: [12] iter= 6890, img_loss=0.031328, reg_loss=0.014355, lr=0.00084, time=340.5 ms, rem=17.65 m free gpu memory: [12] iter= 6900, img_loss=0.032246, reg_loss=0.014353, lr=0.00083, time=341.8 ms, rem=17.66 m free gpu memory: [12] iter= 6910, img_loss=0.062743, reg_loss=0.014348, lr=0.00083, time=348.9 ms, rem=17.97 m free gpu memory: [12] iter= 6920, img_loss=0.041931, reg_loss=0.014357, lr=0.00083, time=339.1 ms, rem=17.41 m free gpu memory: [12] iter= 6930, img_loss=0.036670, reg_loss=0.014351, lr=0.00082, time=342.8 ms, rem=17.54 m free gpu memory: [12] iter= 6940, img_loss=0.041322, reg_loss=0.014353, lr=0.00082, time=341.5 ms, rem=17.42 m free gpu memory: [12] iter= 6950, img_loss=0.069085, reg_loss=0.014352, lr=0.00081, time=342.9 ms, rem=17.43 m free gpu memory: [12] iter= 6960, img_loss=0.044809, reg_loss=0.014355, lr=0.00081, time=342.1 ms, rem=17.33 m free gpu memory: [12] iter= 6970, img_loss=0.012372, reg_loss=0.014369, lr=0.00081, time=338.1 ms, rem=17.07 m free gpu memory: [12] iter= 6980, img_loss=0.072447, reg_loss=0.014340, lr=0.00080, time=341.5 ms, rem=17.19 m free gpu memory: [12] iter= 6990, img_loss=0.026434, reg_loss=0.014360, lr=0.00080, time=339.9 ms, rem=17.05 m free gpu memory: [12] iter= 7000, img_loss=0.049840, reg_loss=0.014356, lr=0.00080, time=341.3 ms, rem=17.07 m free gpu memory: [12] iter= 7010, img_loss=0.017504, reg_loss=0.014352, lr=0.00079, time=343.2 ms, rem=17.10 m free gpu memory: [12] iter= 7020, img_loss=0.052149, reg_loss=0.014352, lr=0.00079, time=340.3 ms, rem=16.90 m free gpu memory: [12] iter= 7030, img_loss=0.011003, reg_loss=0.014367, lr=0.00078, time=339.5 ms, rem=16.81 m free gpu memory: [12] iter= 7040, img_loss=0.044049, reg_loss=0.014356, lr=0.00078, time=341.0 ms, rem=16.82 m free gpu memory: [12] iter= 7050, img_loss=0.057942, reg_loss=0.014340, lr=0.00078, time=342.6 ms, rem=16.85 m free gpu memory: [12] iter= 7060, img_loss=0.069860, reg_loss=0.014344, lr=0.00077, time=339.4 ms, rem=16.63 m free gpu memory: [12] iter= 7070, img_loss=0.037822, reg_loss=0.014348, lr=0.00077, time=341.3 ms, rem=16.67 m free gpu memory: [12] iter= 7080, img_loss=0.044715, reg_loss=0.014334, lr=0.00077, time=341.9 ms, rem=16.64 m free gpu memory: [12] iter= 7090, img_loss=0.051825, reg_loss=0.014349, lr=0.00076, time=342.6 ms, rem=16.62 m free gpu memory: [12] iter= 7100, img_loss=0.067228, reg_loss=0.014339, lr=0.00076, time=341.3 ms, rem=16.50 m free gpu memory: [12] iter= 7110, img_loss=0.061046, reg_loss=0.014344, lr=0.00076, time=344.6 ms, rem=16.60 m free gpu memory: [12] iter= 7120, img_loss=0.012341, reg_loss=0.014358, lr=0.00075, time=341.8 ms, rem=16.41 m free gpu memory: [12] iter= 7130, img_loss=0.042959, reg_loss=0.014350, lr=0.00075, time=341.9 ms, rem=16.35 m free gpu memory: [12] iter= 7140, img_loss=0.036931, reg_loss=0.014347, lr=0.00075, time=342.4 ms, rem=16.32 m free gpu memory: [12] iter= 7150, img_loss=0.030748, reg_loss=0.014354, lr=0.00074, time=336.0 ms, rem=15.96 m free gpu memory: [12] iter= 7160, img_loss=0.023608, reg_loss=0.014342, lr=0.00074, time=339.2 ms, rem=16.06 m free gpu memory: [12] iter= 7170, img_loss=0.056848, reg_loss=0.014344, lr=0.00074, time=338.5 ms, rem=15.97 m free gpu memory: [12] iter= 7180, img_loss=0.019829, reg_loss=0.014355, lr=0.00073, time=337.1 ms, rem=15.84 m free gpu memory: [12] iter= 7190, img_loss=0.050579, reg_loss=0.014338, lr=0.00073, time=340.0 ms, rem=15.92 m free gpu memory: [12] Traceback (most recent call last): File "E:\test4video\nvdiffrec\train.py", line 595, in geometry, mat = optimize_mesh(glctx, geometry, mat, lgt, dataset_train, dataset_validate, File "E:\test4video\nvdiffrec\train.py", line 422, in optimize_mesh total_loss.backward() File "C:\Users\Matt\anaconda3\envs\dmodel\lib\site-packages\torch\_tensor.py", line 363, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "C:\Users\Matt\anaconda3\envs\dmodel\lib\site-packages\torch\autograd\__init__.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: CUDA out of memory. Tried to allocate 140.00 MiB (GPU 0; 8.00 GiB total capacity; 4.49 GiB already allocated; 0 bytes free; 5.55 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF ```

It fails to reach halfway through training during this test, and without changing other settings, will continue training below the number of iters that it OOM's at. For this case, at 6k iters the config will complete the model(with the same issue I posted in #14 ), but 10k iters cannot. This is the main limit on getting a batchsize of 2 or more to improve some training. (in addition to using 512 square images and fewer images overall with some filtering; a larger batchsize is possible on 8gb, if this OOM didn't limit the maximum iters at higher batch sizes than 1)

output when running nvdia-smi inside the iter update loop

def get_gpu_memory():
    command = "nvidia-smi --query-gpu=memory.free --format=csv"
    memory_free_info = sp.check_output(command.split()).decode('ascii').split('\n')[:-1][1:]
    memory_free_values = [int(x.split()[0]) for i, x in enumerate(memory_free_info)]
    return memory_free_values

[JHnvidia]

Thanks @mjc619. I can confirm I see the same on my side. I'm not sure how long it will take to fix, but after some experimenting it seems that the memory leak happens every time we dump a validation frame. As a short term workaround you could increase the save_image / display_image parameters in the config to some large value (e.g. 1-2k) and it should hopefully reduce the memory load.

[jmunkberg]

Thanks for reporting this issue @mjc619 ! We have hopefully fixed the issue now (itertools.cycle leaked memory). Pushed in CL https://github.com/NVlabs/nvdiffrec/commit/3e7007ca0f504008e89eb9a46907cf39ed166117

[not-william]

Hello, would it be correct to adjust mesh_scale for custom datasets by inspecting the progress images, and choosing the smallest value that keeps the whole object in the frame?

I am getting poor results on my own data, and I'm trying to establish if the cause is blurred images, poor masks, incorrect camera poses or incorrect nvdiffrec config.

[jmunkberg]

Hello @not-william

Yes, as stated earlier in the thread:

You can either increase the mesh scale parameter in the .json config "mesh_scale" : 2.5, (use a value that tightly fits your model) or add a scaling to your transform matrix in the dataset. We start from a tetrahedral grid, and if that grid is smaller than the model to capture, the result can be cut as shown above.

We usually do this by visual inspection. The mesh scale only affects the size of the tetrahedral grid. You can see if that is too small by looking at the images early in training. If the object is too large, and does not fit on screen, you need to scale your transform matrix in the dataset loader. I would recommend to scale your transform matrix so that the entire object is visible on screen.

We are fairly sensitive to blurred images, poor masks, and incorrect camera poses. We studied robustness to mask corruption in the supplemental material of the paper and it seems to work fairly well. We do not correct for noisy/incorrect poses, so if they are incorrect, it won't train.

[bigian98]

Hi @Sazoji ! Is this script still available? I would like to run the code on my own data; I already have the images, the masks obtained with detectron2 (0/1 values) and the camera poses (.npy) obtained with Colmap's img2poses. I would like to understand what I am missing in order to proceed. Thank you!

[Sazoji]

Hi @Sazoji ! Is this script still available? I would like to run the code on my own data; I already have the images, the masks obtained with detectron2 (0/1 values) and the camera poses (.npy) obtained with Colmap's img2poses. I would like to understand what I am missing in order to proceed. Thank you!

oh, I'm currently looking at the ways for normal NeRF datasets to handle multiple resolutions, but the links to the scripts have changed since I swapped my username https://gist.github.com/Sazoji/e20835d652c51f305ce328342af7fefd and https://github.com/Sazoji/nvdiffrec/blob/main/colmap2poses.py has the scripts to make a simple dataset off your own images, but it works best with manually filtered video/select images and still holds a lot of refactored code from instant-ngp's colmap2nerf if you already have masked images you just dont pass the mask commands. Use pillow or opencv to make your detectron masks into a separate images with identical names to your source images, following the LLFF/NeRD format, download the dataset and be sure to follow the file structure(/data/your_dataset/images/XXXX.png and /data/your_dataset/mask/XXXX.png).

tl;dr you dont need --json since it's following the LLFF dataset and since you're using your own mask dont pass --mask into the colmap2poses.py if you dont have colmap installed, pass --colmap_path

[mexicantexan]

@Sazoji These were the steps I have taken this far:

1. Structure Dataset
2. Get poses from colmap2poses.py
3. ???
4. 3D model fun times

I guess I'm stuck on what should I do next now that I have the dataset structured and the poses generated from LLFF. Do I need to create my own json file inside of the config folder that has params similar to what is in the neRD json files?

[Sazoji]

@mexicantexan there's a --json option for blender-NeRF format, but you dont need that for nvdiffrec neRD/LLFF datasets use a poses.npy

if you have all these files now, you just need a mask folder for nvdiffrec, there's also the view_imgs.txt which shows what images are used in the dataset, it's a good idea to delete the unused images, before masking ones you don't need to. The rembg masking (within colmap2poses as --mask) is ok, but it's probably better to manually make a crude mask and use CascadePSP, which also can be automated in python.

The --json format is useful for other repos like inistant-ngp and other NeRF repos, but for object capture in nvdiffrec you'd have to apply an alpha layer with the mask to all your images.

[mexicantexan]

@Sazoji My fault, I should have been more specific. You are the only person in this issue section that seems to have ran this successfully on your own dataset, so I applaud you!

Context:

• First I structured my dataset to run through your awesome colmap2poses script
• When running your colmap2poses script I made a change to allow me to pass in my mask directory, because I have my own manually edited masks:

ORIGINAL:

feature_extractor_args = [
            colmap_path, 'feature_extractor',
            '--database_path', os.path.join(basedir, args.database_db),
            '--image_path', os.path.join(basedir, args.images),
            '--ImageReader.single_camera', '1',
            # '--SiftExtraction.use_gpu', '0',
        ]

CHANGED TO:

feature_extractor_args = [
            colmap_path, 'feature_extractor',
            '--database_path', os.path.join(basedir, args.database_db),
            '--image_path', os.path.join(basedir, args.images),
            '--ImageReader.single_camera', '1',
            '--ImageReader.mask_path', mask_dir,
            # '--SiftExtraction.use_gpu', '0',
        ]

• Once I ran colmap on my dataset and have generated the poses_bound.npy file, my current folder structure looks like this:
• The folder images contains 502 jpg images sized at 2000x2000 and with exif data included.
• The folder masks contains 502 manually edited png images sized at 2000x2000 and are named exactly the same as the jpgs in the images folder. These are my masks for the images folder, I never used rembg, instead I just passed the mask folder into colmap with the --ImageReader.mask_path flag
• The images in both folders are in uint8 data format.

Specific Questions:

1. In every readme in the nerd dataset, they resize the images to 512x512. Will resizing affect my dataset or should I keep my images at their current size of 2000x2000?
2. If I do resize the images do I assume that I need to rerun the colmap2poses script, would that be true?
3. Are the steps I am taking so far mimmick what you did?
4. Lastly and most importantly, what would the command line argument for running nvdiffrec on my generated dataset look like/what command line argument did you use?

If you need any more info from me feel free to ask, happy to share whatever is needed to get this working and hopefully help others along the way

[Sazoji]

1 Image size correlates to more vram cost to load everything, I've been able to use 800p for instance, but if you had a high VRAM GPU you'd just want to change the batch size and make sure it trains in enough time. 2 seems like NVIDIA already resizes the NeRD datasets without re-mapping already, editing the images as a whole is fine, like changing the exposure. 3 For the eagle I took around 55 photos without checking for exposure/zebra around the wings, deleted 5, resized to 800pixels per image for instant-ngp, ran rembg as a rough mask and made the first LLFF dataset with colmap2poses, manually cleaned up some of the wings and still don't really like the noise in the highlights. For the salt shaker, I used video set at 500 initially, but set 800 in the colmap2poses script once I could get a higher batch size at that resolution without crashing a few hours in to a leak. Comparing the results showed how much getting better lighting helps with brighter areas. After that, I've been trying to get microscopes to capture insects occasionally, but I need a better way to get longer exposures and better lighting.

4 oh yea there is another .json file you need to make, but it's not really part of the dataset since it's config options for nvidffrec. I copied the ehead config and adjusted the res to the image size and batch settings to fit my vram. Laplace_scale and a couple other settings I added to see what affected the training for the salt shaker, but the biggest contribution I saw was adjusting the learning rate to the batch size and making sure the whole model was within the mesh size. Many of those settings seem to affect how the lighting is captured, and I notice that it works best if the textures aren't becoming really noisy (sort of peppery, with a lot of specs in the light texture) after the first few iters that the settings should work better. Things like changing the GPU memory clock affects those textures, but learning rate is the main dial on changing that.

[Sazoji]

💀

[mexicantexan]

Alright so for anyone trying to run this on their own datasets here was my final step-by-step instruction on getting this to run on Windows. With scripts!!!

Assumptions:

• you have a set of images of at least 50 images that capture all sides of the object you are trying to test on
• you have the correlating masks to each of the images above that have the background as black and the foreground/subject as white
• images are all the same size and the same name
• you are using the LLFF/colmap approach
• Python version is Python3.9
• Windows OS

Steps:

1. Get this repo running. Ensure that the spot/bob training sets train/render without error before going on with step 2.
2. Using @Sazoji colmap2poses.py script. Around line 130 you will see:

   feature_extractor_args = [
           colmap_path, 'feature_extractor',
           '--database_path', os.path.join(basedir, args.database_db),
           '--image_path', os.path.join(basedir, args.images),
           '--ImageReader.single_camera', '1',
           # '--SiftExtraction.use_gpu', '0',
       ]

   If you want to pass in your own masks, change that to:

   feature_extractor_args = [
           colmap_path, 'feature_extractor',
           '--database_path', os.path.join(basedir, args.database_db),
           '--image_path', os.path.join(basedir, args.images),
           '--ImageReader.single_camera', '1',
           '--ImageReader.mask_path', {{INSERT ABSOLUTE PATH TO YOUR MASK DIRECTORY}},
           # '--SiftExtraction.use_gpu', '0',
       ]

3. Run the colmap2poses script so that you get an output that looks like:
4. For this repo to even have an attempt at being able to run properly check that in the view_imgs.txt there are a sufficient number of images. One or two will likely not cut it.
5. remove the images that are not in the view_imgs.txt. You can use a script similar to this:

   base_dir = {{ABSOLUTE PATH TO PARENT DIRECTORY}}
   usable_files_list = []
   defect_directory = os.path.join(base_dir, 'unused_images').replace('\\', '/')
   defect_image_directory = os.path.join(defect_directory, 'images').replace('\\', '/')
   defect_mask_directory = os.path.join(defect_directory, 'mask').replace('\\', '/')
   dirs = [defect_directory, defect_image_directory, defect_mask_directory]
   for d in dirs:
   if not os.path.isdir(d):
       os.makedirs(d)
   with open(os.path.join(base_dir, 'view_imgs.txt')) as fp:
   for line in fp:
       usable_files_list.append(line.strip())
   temp_all_img = []
   temp_all_mask = []
   for idx, i in enumerate(all_img):
   if os.path.split(i)[1] not in usable_files_list:
       try:
           shutil.move(i, defect_image_directory)
       except shutil.Error:
           os.unlink(i)
   else:
       temp_all_img.append(i)
   for idx, i in enumerate(all_mask):
   if os.path.split(i)[1] not in usable_files_list:
       try:
           shutil.move(i, defect_mask_directory)
       except shutil.Error:
           os.unlink(i)
   else:
       temp_all_mask.append(i)
   all_img = temp_all_img
   all_mask = temp_all_mask

6. Copy the parent directory into the neRD dataset folder ( nvdiffrec/data/nerd) and name it something unique.
7. Inside nvdiffrec/data/nerd there is a scale_images.py script. Run it on your new dataset.
8. Inside of nvdiffrec/configs make a copy of the nerd_ehead.json and name it something unique.
9. Inside of the file you just copied adjust the ref_mesh and out_dir at a minimum, and adjust other args as you need to. I was able to get this running on my Titan RTX's with 24GB of VRAM, but if you have an 8GB card, it likely won't work well tbh.

[neerajtiwari360]

Hi @Sazoji ! Is this script still available? I would like to run the code on my own data; I already have the images, the masks obtained with detectron2 (0/1 values) and the camera poses (.npy) obtained with Colmap's img2poses. I would like to understand what I am missing in order to proceed. Thank you!

Hi, did you get any solution for it? please share i got stuck at same place

[Sazoji]

@neerajtiwari360 the script is now here since I updated my username, I believe I linked it earlier in the thread. Apple has released a better code base and scripts for translating and preparing NeRF/LLFF data, but it includes a lot of data processing exclusively for their repo, so I may reimplement the script to handle video with their more readable code. Probably when a new repo that accelerates the texture ray approaches releases. If you already have a pose file, images and generated masks, you just need the dataset structure to match the NeRD/LLFF datasets so nvdiffrec can read the files.

[DemainTrust]

I've been able to run this on my own data (slowly, on a 3070) using U^2Net matting and colmap2nerf code from instant-ngp (if others are following this, remove the image extensions, and it should run on windows) The issue I'm having is val, and test data or the lack of it. Are there config options to remove the requirement for those datasets. Copying and renaming the json lists to val and train works, but is rather cumbersome, and I was wondering if I was missing a better option with the NeRD dataset preparation, which use manual masks not applied to the image itself (which is possible with U^2 net, and would help with colmap mapping since the background can still be used for feature tracking.)

I haven't looked into what data is required to be presented in the config, nor resolution/training options, but I'm just wondering what's the generally intended arguments and method for presenting your own data, when no val is available.

how u do that? i am noob this...

can you some explain how to make your own data model??

[IvanGarcia7]

@Sazoji @mexicantexan When I try to run colmap2poses on a dataset (350 images), no error appears, but if I look at the view_imgs.txt file only 4 images are shown.

Am I doing something wrong?

Thanks in advance.

[Sazoji]

When I try to run colmap2poses only 4 images are shown.

Colmap takes what images it can map together, try exhaustive mapping or reduce image resolution to avoid some motion blur.

Under absolutely perfect conditions, colmap shouldn't reject any images, but if you're feeding a large unfiltered dataset it's better to spend the extra time in exhaustive mapping to check each image against each other. Changing res or manually using only images with as little movement of the subject/background should help. I also should note interpolation on video to increase the number of images in a dataset doesn't improve the quality of the model even when the interpolated images are added to the set, tried that for NeRFs awhile ago.

[IvanGarcia7]

When I try to run colmap2poses only 4 images are shown.

Colmap takes what images it can map together, try exhaustive mapping or reduce image resolution to avoid some motion blur.

Under absolutely perfect conditions, colmap shouldn't reject any images, but if you're feeding a large unfiltered dataset it's better to spend the extra time in exhaustive mapping to check each image against each other. Changing res or manually using only images with as little movement of the subject/background should help. I also should note interpolation on video to increase the number of images in a dataset doesn't improve the quality of the model even when the interpolated images are added to the set, tried that for NeRFs awhile ago.

I recorded a video, so the images are sequentially arranged. The camera is held at a fixed point while the object slowly rotates on a platform.

[Sazoji]

hmm, try using rembg before colmap, so it won't try to map the camera's position based on the background, I dont know if nvdiffrec would like it since it's trying to reconstruct HDRI lighting based on camera position.

I know when creating NeRFs back last febuary, I had success that way to remove some noise and map more images. Exhaustive mode changes the Big O computation time depending on the number of images (but might help to get more images accepted), so try masking before messing with that.

[IvanGarcia7]

hmm, try using rembg before colmap, so it won't try to map the camera's position based on the background, I dont know if nvdiffrec would like it since it's trying to reconstruct HDRI lighting based on camera position.

I know when creating NeRFs back last febuary, I had success that way to remove some noise and map more images. Exhaustive mode changes the Big O computation time depending on the number of images (but might help to get more images accepted), so try masking before messing with that.

Perfect, thank you very much for the advice, I'll try it.

[prakashknaikade]

Alright so for anyone trying to run this on their own datasets here was my final step-by-step instruction on getting this to run on Windows. With scripts!!!

Assumptions:

* you have a set of images of at least 50 images that capture all sides of the object you are trying to test on

* you have the correlating masks to each of the images above that have the background as black and the foreground/subject as white

* images are all the same size and the same name

* you are using the LLFF/colmap approach

* Python version is Python3.9

* Windows OS

Steps:

1. Get this repo running. Ensure that the spot/bob training sets train/render without error before going on with step 2.

2. Using @Sazoji `colmap2poses.py` [script](https://github.com/Sazoji/nvdiffrec/blob/main/colmap2poses.py). Around line 130 you will see:

feature_extractor_args = [
            colmap_path, 'feature_extractor',
            '--database_path', os.path.join(basedir, args.database_db),
            '--image_path', os.path.join(basedir, args.images),
            '--ImageReader.single_camera', '1',
            # '--SiftExtraction.use_gpu', '0',
        ]

If you want to pass in your own masks, change that to:

feature_extractor_args = [
            colmap_path, 'feature_extractor',
            '--database_path', os.path.join(basedir, args.database_db),
            '--image_path', os.path.join(basedir, args.images),
            '--ImageReader.single_camera', '1',
            '--ImageReader.mask_path', {{INSERT ABSOLUTE PATH TO YOUR MASK DIRECTORY}},
            # '--SiftExtraction.use_gpu', '0',
        ]

3. Run the colmap2poses script so that you get an output that looks like:
   ![image](https://user-images.githubusercontent.com/25260617/175576255-08592326-c4a9-4dd4-b98b-62ca40461117.png)

4. For this repo to even have an attempt at being able to run properly check that in the `view_imgs.txt` there are a sufficient number of images. One or two will likely not cut it.

5. remove the images that are not in the `view_imgs.txt`. You can use a script similar to this:

base_dir = {{ABSOLUTE PATH TO PARENT DIRECTORY}}
usable_files_list = []
defect_directory = os.path.join(base_dir, 'unused_images').replace('\\', '/')
defect_image_directory = os.path.join(defect_directory, 'images').replace('\\', '/')
defect_mask_directory = os.path.join(defect_directory, 'mask').replace('\\', '/')
dirs = [defect_directory, defect_image_directory, defect_mask_directory]
for d in dirs:
    if not os.path.isdir(d):
        os.makedirs(d)
with open(os.path.join(base_dir, 'view_imgs.txt')) as fp:
    for line in fp:
        usable_files_list.append(line.strip())
temp_all_img = []
temp_all_mask = []
for idx, i in enumerate(all_img):
    if os.path.split(i)[1] not in usable_files_list:
        try:
            shutil.move(i, defect_image_directory)
        except shutil.Error:
            os.unlink(i)
    else:
        temp_all_img.append(i)
for idx, i in enumerate(all_mask):
    if os.path.split(i)[1] not in usable_files_list:
        try:
            shutil.move(i, defect_mask_directory)
        except shutil.Error:
            os.unlink(i)
    else:
        temp_all_mask.append(i)
all_img = temp_all_img
all_mask = temp_all_mask

6. Copy the parent directory into the neRD dataset folder ( `nvdiffrec/data/nerd`) and name it something unique.

7. Inside `nvdiffrec/data/nerd` there is a `scale_images.py` script. Run it on your new dataset.

8. Inside of `nvdiffrec/configs` make a copy of the `nerd_ehead.json` and name it something unique.

9. Inside of the file you just copied adjust the `ref_mesh` and `out_dir` at a minimum, and adjust other args as you need to. I was able to get this running on my Titan RTX's with 24GB of VRAM, but if you have an 8GB card, it likely won't work well tbh.

Thank you for detailed steps. Can you please add steps to handle assumptions you mentioned like how to create masks (you have the correlating masks to each of the images above that have the background as black and the foreground/subject as white)?

Also, why colmap fails to calculate poses for all the images in the folder?

[prakashknaikade]

Hello @Sazoji @mexicantexan ,

I got masks from the scrpit: https://github.com/Sazoji/nvdiffrec/blob/main/colmap2poses.py, but while scaling images using scale_images.py, I am getting this error or mask images,

primaries_14_aug_2/masks/0228.jpg
Traceback (most recent call last):
  File "/HPS/ColorNeRF/work/nvdiffrec/data/scale_images.py", line 33, in <module>
    img = img[None, ...].permute(0, 3, 1, 2)
RuntimeError: permute(sparse_coo): number of dimensions in the tensor input does not match the length of the desired ordering of dimensions i.e. input.dim() = 3 is not equal to len(dims) = 4

please help.