Progressive_view, batch_sizes can also change with resolution_milestones

[voletiv]

1. Makes progressive increase of elevation, azimuth

Just set progressive_until steps in the config file. elevation_range, azimuth_range are progressively (linearly) increased from their default eval values to the full range values over progressive_until steps. If progressive_until = 0, no progression is made.

2. Can set batch_size as a list for different resolutions. For e.g.,

   height: [64, 512]
   width: [64, 512]
   batch_size: [12, 3]
   resolution_milestones: [1000]

   In this case, before iter1000, training is at 64x64 with batch_size 12, after iter1000, training is at 512x512 with batch_size 3.

[claforte]

Thanks Vikram!

This introduces a new convention and doesn't provide much control over the rate of expansion. Can't we instead reuse the existing convention, e.g. min_step_percent: [0, 0.4, 0.02, 2000] # (start_iter, start_val, end_val, end_iter)?

[claforte]

BTW I'm not sure when radius or fov range expansion would be desirable, and if so, whether we'd start from the larger value, or the smaller one. Either way, if we reuse the old convention, we'll have more flexibility in this regard as well.

[bennyguo]

I agree with Christian. It would be better if we could use something like self.C to update arbitrary values in the dataset. However in the dataset we currently are not able to access the global step, any thoughts?

[bennyguo]

And we probably need a more flexible expansion instead of the linear interpolation of the current self.C. For example, in ProlificDreamer, we increase the render resolution at certain steps.

[voletiv]

1. The problem indeed was not being able to access global_step in the data, I "fixed" that by making data.collate fn record its own step.
2. radius/fovy start from the eval value, not the higher/lower ones. This is how it was in stable dreamfusion as well So it's effectively like the (start_iter, start_val, end_val, end_iter) scheme. For example, the fovy range's start and end values change like so: (0, cfg.eval_fovy, cfg.fovy_range[0], cfg.progressive_until) (0, cfg.eval_fovy, cfg.fovy_range[1], cfg.progressive_until)

[bennyguo]

Let the dataset maintain its state can be problematic when resuming from checkpoints as the state is not saved to the checkpoint. How about setting the global step in update_step?

[voletiv]

True, I wanted to implement progressive_view within update_step initially (check https://github.com/threestudio-project/threestudio/pull/165/commits/4cf6f1b84d7ad0bb74b76cc634b0ce02a324ef51), but I noticed that update_step is never called anywhere. Can you point to how it could be used?

[bennyguo]

You have to make the dataset inherit to Updateable for this to work. See the uncond dataset for an example. One thing to notice is that you also need to disable multi-processing data loading by setting num_workers to 0, or you won't be able to modify self attributes of the dataset (the reason for this is not clear).

[claforte]

@voletiv I'd like to try this, this week-end... please include in your description on top an example command line for the 2nd phase of Anya, and a typical result you get (if you have that handy). Thanks!

[voletiv]

My typical command for it is: python launch.py --config configs/experimental/imagecondition_zero123nerf.yaml --train --gpu 0 system.prompt_processor.prompt="A DSLR 3D photo of a cute anime schoolgirl stands proudly with her arms in the air, pink hair ( unreal engine 5 trending on Artstation Ghibli 4k )" system.weights=outputs/zero123/128_anya_front_rgba.png@20230623-145711/ckpts/last.ckpt system.freq.guidance_eval=13 system.loggers.wandb.enable=true system.loggers.wandb.project="voletiv-anya-new" data.image_path=./load/images/anya_front_rgba.png system.loggers.wandb.name="anya" data.random_camera.progressive_until=500

The important part is the last command : data.random_camera.progressive_until=500, which means progressively grow the angles in the first 500 iters.

When I want to switch from SD to DF, I comment/uncomment the relevant lines in the yaml file.

[voletiv]

@bennyguo I've added Updateable at the right place, please check!

I've also added changing of batch_size with resolution_milestone, please check!

[bennyguo]

I'll merge this after the comments have been addressed, but we probably need a better way to control these parameters so that we can support more advanced expansion scheme. Will come up with a proposal later.

[voletiv]

Agreed! I didn't want to introduce 4 more parameters (elevation_range_min, elevation_range_max, azimuth_range_min, azimuth_range_max) that could be specified with [start_iter, start_val, end_val, end_iter]. I opted to introduce only one more progressive_until. Perhaps there's a better way to deal with this.

[bennyguo]

Agreed! I didn't want to introduce 4 more parameters (elevation_range_min, elevation_range_max, azimuth_range_min, azimuth_range_max) that could be specified with [start_iter, start_val, end_val, end_iter]. I opted to introduce only one more progressive_until. Perhaps there's a better way to deal with this.

Yes this is what I thought. Using 4 more parameters does not sound so good ...