render novel_pose when using train_stage=2

[Xzy765039540]

How to render novel_pose when using train_stage=2? Can you provide a script? Best Regards,

[wyiguanw]

hi, i met same issue. and I try to modify render_novel_pose.py for stage 2, here is the result:

but the visualization during the training is fine:

this script works perfectly for stage 1, but not for stage 2. and the only different thing that I found is stage 2 have a pose_featmap so, Im not sure why this happened. could you provide script for stage 2 rendering

[Xzy765039540]

hi, i met same issue. and I try to modify render_novel_pose.py for stage 2, here is the result:

but the visualization during the training is fine:

this script works perfectly for stage 1, but not for stage 2. and the only different thing that I found is stage 2 have a pose_featmap so, Im not sure why this happened. could you provide script for stage 2 rendering

I think the main reason should be that the pose_featmap is not loaded when the model is loaded, I slove this problem by add self.pose_encoder.load_state_dict(saved_model_state["pose_encoder"], strict=False) when use stage 2 , I modified the load() function in avatar_model.py def load(self, iteration, test=False):

    net_save_path = os.path.join(self.model_path, "net/iteration_{}".format(iteration))
    if self.model_parms.train_stage  ==1:
        saved_model_state = torch.load(
            os.path.join(net_save_path, "net.pth"))
        print('load pth: ', os.path.join(net_save_path, "net.pth"))
        self.net.load_state_dict(saved_model_state["net"], strict=False)

    if self.model_parms.train_stage  ==2:
        saved_model_state = torch.load(
            os.path.join(net_save_path, "pose_encoder.pth"))
        print('load pth: ', os.path.join(net_save_path, "pose_encoder.pth"))
        self.net.load_state_dict(saved_model_state["net"], strict=False)

    if self.model_parms.train_stage  ==1:
        if not test:
            self.pose.load_state_dict(saved_model_state["pose"], strict=False)
            self.transl.load_state_dict(saved_model_state["transl"], strict=False)
        # if self.train_mode == 0:
        self.geo_feature.data[...] = saved_model_state["geo_feature"].data[...]

    if self.model_parms.train_stage  ==2:
        if not test:
            self.pose.load_state_dict(saved_model_state["pose"], strict=False)
            self.transl.load_state_dict(saved_model_state["transl"], strict=False)
        # if self.train_mode == 0:
        self.geo_feature.data[...] = saved_model_state["geo_feature"].data[...]
        self.pose_encoder.load_state_dict(saved_model_state["pose_encoder"], strict=False)

    if self.optimizer is not None:
        self.optimizer.load_state_dict(saved_model_state["optimizer"])
    if self.scheduler is not None:
        self.scheduler.load_state_dict(saved_model_state["scheduler"])

[Xzy765039540]

May I ask what the main improvements were made in stage 2, because I've noticed that there's not much difference in the rendering results between stage 1 and stage 2.

[serizawa-04013958]

I faced same problems. How can i render with stage2? my stage1 results with neuman dataset was not so good.

[2019211001001009-XieYueCong]

嗨，我遇到了同样的问题。我尝试修改第 2 阶段，结果如下：render_novel_pose.py 但是训练期间的可视化效果很好： 此脚本适用于第 1 阶段，但不适用于第 2 阶段。我发现的唯一不同之处在于第 2 阶段有一个pose_featmap所以，我不确定为什么会这样。您能否提供第 2 阶段渲染的脚本

我认为主要原因应该是加载模型时没有加载pose_featmap，我在使用阶段 2 时通过添加 self.pose_encoder.load_state_dict（saved_model_state[“pose_encoder”]， strict=False） 来解决这个问题，我修改了 avatar_model.py def load（self， iteration， test=False） 中的 load（） 函数：

    net_save_path = os.path.join(self.model_path, "net/iteration_{}".format(iteration))
    if self.model_parms.train_stage  ==1:
        saved_model_state = torch.load(
            os.path.join(net_save_path, "net.pth"))
        print('load pth: ', os.path.join(net_save_path, "net.pth"))
        self.net.load_state_dict(saved_model_state["net"], strict=False)

    if self.model_parms.train_stage  ==2:
        saved_model_state = torch.load(
            os.path.join(net_save_path, "pose_encoder.pth"))
        print('load pth: ', os.path.join(net_save_path, "pose_encoder.pth"))
        self.net.load_state_dict(saved_model_state["net"], strict=False)

    if self.model_parms.train_stage  ==1:
        if not test:
            self.pose.load_state_dict(saved_model_state["pose"], strict=False)
            self.transl.load_state_dict(saved_model_state["transl"], strict=False)
        # if self.train_mode == 0:
        self.geo_feature.data[...] = saved_model_state["geo_feature"].data[...]

    if self.model_parms.train_stage  ==2:
        if not test:
            self.pose.load_state_dict(saved_model_state["pose"], strict=False)
            self.transl.load_state_dict(saved_model_state["transl"], strict=False)
        # if self.train_mode == 0:
        self.geo_feature.data[...] = saved_model_state["geo_feature"].data[...]
        self.pose_encoder.load_state_dict(saved_model_state["pose_encoder"], strict=False)

    if self.optimizer is not None:
        self.optimizer.load_state_dict(saved_model_state["optimizer"])
    if self.scheduler is not None:
        self.scheduler.load_state_dict(saved_model_state["scheduler"])

Have you solved it? How to render new postures in the second stage after training