search

ryanhe312 / STSSNet-AAAI2024

Low-latency Space-time Supersampling for Real-time Rendering
MIT License
27 stars 1 forks source link

Whether the network is generalizable #1

Open ss0912 opened 3 months ago

ss0912 commented 3 months ago

I noticed that the checkpoints you provide contain the pkl file for each scenario, so do I need to retrain once for each scenario?

ryanhe312 commented 3 months ago

Not necessarily. Our experiments are two-fold: we evaluate for single-scene (train and test on LE, ST, and SW separately) and cross-scene (train on LE, ST, and SW, test on AR) settings. The experiments on AR show the generalization capability of our model.

ss0912 commented 3 months ago

Thank you very much for your answer. I would like to ask if your network can be used in UE? Just like DLSS.

ss0912 commented 3 months ago

I would also like to ask, how do you handle your network dataset?

ryanhe312 commented 3 months ago

Thank you very much for your answer. I would like to ask if your network can be used in UE? Just like DLSS.

Currently not available. We are still working on it.

I would also like to ask, how do you handle your network dataset?

Do you mean dataset storage? We store the data as npy files.

ss0912 commented 3 months ago

I mean how to process the images collected from unreal engine into npy files. I noticed that you only provide npy files for a few pre-processed scenes. If I want to train my own collected scenes, how should I process my image data?

ryanhe312 commented 3 months ago

You may refer to ExtraNet for the detailed data processing pipeline. Npy files are just for compressing.

ss0912 commented 2 months ago

I have a question regarding "STSSNet". As I understand it, the input to STSSNet includes information from frames at t-5, t-3, and t-1, and the output is the prediction for frame t. Is my understanding correct?

ryanhe312 commented 2 months ago

For extrapolation frames, yes. For super-resolution frames, inputs are t-4, t-2, and t. Please refer to the paper for details.

ss0912 commented 1 month ago

I noticed that you have the training and test datasets in a compressed format. I know that in "compressedHR.XXXX.npy" is saved the content of the image in high resolution. I would like to know what exactly is saved in the "compressed.XXXX.NoWarp.npy" and "compressed.XXXX.Warp.npy" files in the NPY folder you provided? If you can answer this for me, I would be very grateful.