Cannot estimate normal map and displacement map and cannot check the output

[j20232]

Thank you for sharing the code.

I tried it and fixed some bugs like belows because your code doesn't work.

Fixed bugs

facialDetails.py: around line 39

def load_image(img_path, target_size):
    img = Image.load(img_path).convert('LA')
    if img is None:
        print(img_path)
    im = np.asarray(img.resize(img, target_size,Image.ANTIALIAS))
    # im = np.asarray(img.resize(img, target_size,, Image.ANTIALIAS))

facialDetails.py: around line 128

def loadimage(data root, args):
   img = Image.open(dataroot)
   img = img.resize((args.imageW, args.imageW))
   # img = Image.resize(Image.open(dataroot), (args.imageW, args.imageW))
   ...

facialDetails.py: around line 280

   displacementMap = (displacementMap+1)/2*65535
   displacementMap = displacementMap.astype("uint16")
   array_buffer = displacementMap.tobytes()
   img = Image.new("I", displacementMap.T.shape)
   img.frombytes(array_buffer, "raw", "I;16")
   save_path = os.path.join(args.output_path, base_name[0], 'result.displacementmap.png')
   img.save(save_path)
   # Image.fromarray(displacementMap).save(save_path)

After that, I added dlls not included in your released package to ./textureRender

• glew32.dll
• opencv_world330.dll

Finally, I run the below command, the code worked well other than estimation parts and rendering parts.

python facialDetails.py -i ./samples/details/019615.jpg -o ./results

Error of estimation parts

I think the weight files of DFDN are wrong because the output displacement map and normal map contain many noisy patches caused by not trained networks. Could you check them?

Error of rendering parts

And I cannot check the output with hmrenderer.exe because the material file is wrong. It doesn't contain the normal map and the displacement map.

result.mtl

newmtl FaceTexture
map_Kd result.isomap.png

Moreover, the following part of facialDetails.py doesn't work well.

 if args.visualize:
    args.face_render_path = os.path.abspath(args.face_render_path)
    cmd = '%s/hmrenderer.exe %s %s %s'%(args.face_render_path,save_obj_path+'.obj',save_path,args.face_render_path+'/shaders')

It showed the warning.

LOG (WARN: Material file [ result.mtl ] not found.
WARN: Failed to load material file(s). Use default material.
)
~In file (e:\face2face\code\release_v1\src\face_rendering\src\includes\ind\loaders.cc) , line (81)

I think the path is wrong.

Could check them?

Best Regards,

[apchenstu]

Sorry, I tried to remove cv2 libs and forgot to test the code, thanks for you help, I already fix the bugs you pointed out, please try again :)

If you don't want to re-download the new released package, you may need to

1. re-install BFM2017 with a new version of texture_uv.mat
2. re-download facialDetails.py and proxyPredictor.py.

[ghost]

@apchenstu I get this error Load preTrain model done. Load preTrain model done. Traceback (most recent call last): File "facialDetails.py", line 294, in main(args) File "facialDetails.py", line 237, in main fit_landmarks(args.landmark_exe_path, args.img_path) File "facialDetails.py", line 57, in fit_landmarks os.chdir(landmark_exe_path) FileNotFoundError: [Errno 2] No such file or directory: ./landmarks

[apchenstu]

You may used an old version, this error is due to I renamed some folders. You can try the newest one release package, Thanks.

[ghost]

@apchenstu I used the released package attached, I think the problem is with the exe files as i am trying to setup the project in google colab and currently it is setup to work only in windows Load preTrain model done. Load preTrain model done. sh: 1: FaceLandmarkImg.exe: not found ===> Landmarks detection done.

===> estimating proxy of /content/Facial_Details_Synthesis/src/test.jpg Traceback (most recent call last): File "/usr/lib/python3.6/shutil.py", line 550, in move os.rename(src, real_dst) FileNotFoundError: [Errno 2] No such file or directory: '/content/Facial_DetailsSynthesis/src/landmarkDetector/processed/SH3mfFA.box' -> '/content/Facial_DetailsSynthesis/src/result/SH3mfFA/SH3mfFA_.box'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "facialDetails.py", line 294, in main(args) File "facialDetails.py", line 248, in main move_landmark(args.landmark_exe_path,save_path,base_name[0]) File "facialDetails.py", line 51, in move_landmark shutil.move(os.path.join(landmark_exe_path,'processed',name+'.box'),os.path.join(save_path,name+'.box')) File "/usr/lib/python3.6/shutil.py", line 564, in move copy_function(src, real_dst) File "/usr/lib/python3.6/shutil.py", line 263, in copy2 copyfile(src, dst, follow_symlinks=follow_symlinks) File "/usr/lib/python3.6/shutil.py", line 120, in copyfile with open(src, 'rb') as fsrc: FileNotFoundError: [Errno 2] No such file or directory: '/content/Facial_DetailsSynthesis/src/landmarkDetector/processed/SH3mfFA.box'

[apchenstu]

Yeah, I didn't test the code on other platforms. Your error looks like the landmarkDetector not works well. I will write the compiling guideline in those two days, maybe you can try to re-compile the code in google colab by then. Thanks

[j20232]

Sorry, I tried to remove cv2 libs and forgot to test the code, thanks for you help, I already fix the bugs you pointed out, please try again :)

If you don't want to re-download the new released package, you may need to

1. re-install BFM2017 with a new version of texture_uv.mat
2. re-download facialDetails.py and proxyPredictor.py.

Thank you for your quick response, but there are still many bugs.

1. The output part of the displacement map is wrong.

As I pointed out before, line 280 - 282 in facialDetails.py are wrong. This part outputs the error like belows.

===> predicting details of ...\FacialDetailsSynthesis\samples\details\019615.jpg
Traceback (most recent call last):
  File "...\PIL\PngImagePlugin.py", line 771, in _save
    rawmode, mode = _OUTMODES[mode]
KeyError: 'I;16'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "facialDetails.py", line 297, in <module>
    main(args)
  File "facialDetails.py", line 282, in main
    Image.fromarray(displacementMap.astype('uint16')).save(save_path)
  File "...\PIL\Image.py", line 1994, in save
    save_handler(self, fp, filename)
  File "...\PIL\PngImagePlugin.py", line 773, in _save
    raise IOError("cannot write mode %s as PNG" % mode)
OSError: cannot write mode I;16 as PNG

2. Pretrained weights may be wrong.

The pretrained weights of DFDN may be wrong because the estimation of the normal and distplacement map doesn't work. The outputs contain many block noises.

3. Linking the output material file with the normal and displacement map isn't correct.

result.mtl is always the following format.

newmtl FaceTexture
map_Kd result.isomap.png

It doesn't contain the normal and displacement map.

4. the path of hmrenderer.exe is wrong

Line 291 in facialDetails.py doesn't work well because the renderer cannot find the material file.

cmd = '%s/hmrenderer.exe %s %s %s'%(args.face_render_path,save_obj_path+'.obj',save_path,args.face_render_path+'/shaders')

So, the output of the rendering shows only the mesh and doesn't show the albedo.

I'm sorry many times.

Could you check them?

[apchenstu]

No worries, I noted the bugs you pointed out, actually, these four bugs are the same problem, it is because the code could not correctly load the input image. Maybe we use different version PIL libs(PIL 6.0.0 in my PC), and the anaconda default version is 'PIL 6.1.0' now, I will fix this. Thanks.

[apchenstu]

I just setup the environment and test the code in a new PC, it can correctly output normalMap and displacementMap, for the noises output, can you show me the results, I also got noisy result before I fixed the bugs you point out, it's because value range (i.e., opencv range 0-255 and PIL range 0-1.0 after transform RGB to HSV), please check this line here.

Please ignore material warming as our phong shading model would not use material and texture.

[ghost]

@apchenstu thanks looking forward to compile guide to try to get it to work in google colab

[j20232]

@apchenstu Thank you very much. I could try your project.