search

appleseedhq / appleseed

A modern open source rendering engine for animation and visual effects
https://appleseedhq.net/
MIT License
2.2k stars 330 forks source link

Scene render will not complete frame range, only object animated is the camera. #2676

Closed samhodge closed 5 years ago

samhodge commented 5 years ago

Hi I have a scene in gaffer which I will package for disection at a later stage.

I can render the geometry with a emission shader for the entire frame range 1-97

But with the look dev and IBL environment light applied: sss, diffuse, glossy, emission.

It gets stuck on frame 67 no matter what I do to the near and far clipping planes.

All the inputs are publicly available without royality costs, but might require a little collecting.

https://www.3dscanstore.com/blog/Free-3D-Head-Model (several meshes and high res TGA textures)

http://bit.ly/castle-camera (Alembic Camera at 24fps)

http://bit.ly/castle-HDR-kognat (light probe)

http://bit.ly/castle_plate (need to use FFMPEG to rip to sequence and treat at 24fps, even though the footage is 30fps, 97 frames total)

Would this be of interest to the Appleseed castle_github.gfr.zip HQ developers to have this setup?

The rendered result up to frame 66 from the range 1-97.

wip.mp4.zip

samhodge commented 5 years ago

here is the log pasted in

2019-08-12T11:49:17.629168Z <001>   516 MB info    | rendering frame...
2019-08-12T11:49:17.629354Z <001>   516 MB info    | opening texture file /home/kognat/Downloads/castle.exr and reading metadata...
2019-08-12T11:49:17.633313Z <001>   535 MB info    | setting oiio texture cache size to 953.7 MB.
2019-08-12T11:49:17.633430Z <001>   535 MB info    | setting oiio search path to /opt/gaffer/gaffer-0.53.4.0-linux/appleseedDisplays:/home/kognat/gaffer/oslCode:/opt/gaffer/gaffer-0.53.4.0-linux/shaders:/home/kognat/gaffer/shaders:/opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/gaffer:/opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/appleseed
2019-08-12T11:49:17.633530Z <001>   535 MB info    | setting osl shader search path to /opt/gaffer/gaffer-0.53.4.0-linux/appleseedDisplays:/home/kognat/gaffer/oslCode:/opt/gaffer/gaffer-0.53.4.0-linux/shaders:/home/kognat/gaffer/shaders:/opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/gaffer:/opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/appleseed
2019-08-12T11:49:18.231122Z <001>   549 MB info    | collecting assembly instances...
2019-08-12T11:49:18.238255Z <001>   549 MB info    | building assembly tree (3 assembly instances)...
2019-08-12T11:49:18.258113Z <001>   549 MB info    | collecting geometry for triangle tree #618 from assembly "scene/assembly//group/Brows_Brown_assembly" (1 region)...
2019-08-12T11:49:18.283800Z <001>   572 MB info    | building triangle tree #618 (bvh, 178,560 static triangles, 0 moving triangle)...
2019-08-12T11:49:18.634860Z <001>   595 MB info    | collecting geometry for triangle tree #36 from assembly "scene/assembly" (6 regions)...
2019-08-12T11:49:18.806184Z <001>   682 MB info    | building triangle tree #36 (bvh, 1,279,812 static triangles, 800 moving triangles)...
2019-08-12T11:49:22.030813Z <001>   837 MB info    | collecting light emitters...
2019-08-12T11:49:22.031598Z <001>   837 MB info    | found 0 non-physical light, 1,600 light-tree compatible lights, 1,600 emitting triangles.
2019-08-12T11:49:22.031786Z <001>   837 MB info    | the scene does not rely on alpha mapping and does not contain participating media; using probe tracing.
2019-08-12T11:49:22.031850Z <001>   837 MB info    | rendering effort wasted by tile borders: 6.3% (tile dimensions: 64 x 64, tile margins: 2 x 2)
2019-08-12T11:49:22.034617Z <001>   837 MB info    | generic frame renderer settings:
2019-08-12T11:49:22.034617Z <001>   837 MB info    |   spectrum mode                 rgb
2019-08-12T11:49:22.034617Z <001>   837 MB info    |   sampling mode                 qmc
2019-08-12T11:49:22.034617Z <001>   837 MB info    |   rendering threads             12
2019-08-12T11:49:22.034617Z <001>   837 MB info    |   tile ordering                 spiral
2019-08-12T11:49:22.034617Z <001>   837 MB info    |   passes                        8
2019-08-12T11:49:22.034665Z <001>   837 MB info    | uniform pixel renderer settings:
2019-08-12T11:49:22.034665Z <001>   837 MB info    |   samples                       30
2019-08-12T11:49:22.034665Z <001>   837 MB info    |   force antialiasing            off
2019-08-12T11:49:22.034665Z <001>   837 MB info    |   decorrelate pixels            on
2019-08-12T11:49:22.034665Z <001>   837 MB info    |   diagnostics                   off
2019-08-12T11:49:22.034701Z <001>   837 MB info    | generic sample renderer settings:
2019-08-12T11:49:22.034701Z <001>   837 MB info    |   transparency threshold        0.001000
2019-08-12T11:49:22.034701Z <001>   837 MB info    |   max iterations                100
2019-08-12T11:49:22.034701Z <001>   837 MB info    |   report self intersections     off
2019-08-12T11:49:22.034755Z <001>   837 MB info    | unidirectional path tracer settings:
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   direct lighting               on
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   ibl                           on
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   caustics                      off
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   max bounces                   8
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   max diffuse bounces           3
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   max glossy bounces            8
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   max specular bounces          8
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   max volume bounces            8
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   russian roulette start bounce 6
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   next event estimation         on
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   dl light samples              1.0
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   dl light threshold            0.000
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   ibl env samples               5.0
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   max ray intensity             unlimited
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   volume distance samples       2
2019-08-12T11:49:22.034755Z <001>   837 MB info    |   equiangular sampling          on
2019-08-12T11:49:22.034852Z <001>   837 MB warning | transforms of environment edf "scene//group/light" must be pure rotations but have scaling and/or translation components; these will be ignored.
2019-08-12T11:49:22.034936Z <001>   837 MB info    | building 5376x2688 importance map for environment edf "scene//group/light"...
2019-08-12T11:49:24.989101Z <001>  1306 MB info    | built importance map for environment edf "scene//group/light".
2019-08-12T11:49:24.990718Z <001>  1141 MB info    | camera "scene//group/camera" settings:
2019-08-12T11:49:24.990718Z <001>  1141 MB info    |   model                         pinhole_camera
2019-08-12T11:49:24.990718Z <001>  1141 MB info    |   film width                    3.600000
2019-08-12T11:49:24.990718Z <001>  1141 MB info    |   film height                   2.025000
2019-08-12T11:49:24.990718Z <001>  1141 MB info    |   focal length                  2.800000
2019-08-12T11:49:24.990718Z <001>  1141 MB info    |   near-z                        0.000000
2019-08-12T11:49:24.990718Z <001>  1141 MB info    |   shutter open begin time       66.750000
2019-08-12T11:49:24.990718Z <001>  1141 MB info    |   shutter open end time         66.750000
2019-08-12T11:49:24.990718Z <001>  1141 MB info    |   shutter close begin time      67.250000
2019-08-12T11:49:24.990718Z <001>  1141 MB info    |   shutter close end time        67.250000
2019-08-12T11:49:24.990807Z <001>  1141 MB info    | frame "beauty" settings:
2019-08-12T11:49:24.990807Z <001>  1141 MB info    |   camera                        /group/camera
2019-08-12T11:49:24.990807Z <001>  1141 MB info    |   resolution                    1,920 x 1,080
2019-08-12T11:49:24.990807Z <001>  1141 MB info    |   tile size                     64 x 64
2019-08-12T11:49:24.990807Z <001>  1141 MB info    |   filter                        blackman-harris
2019-08-12T11:49:24.990807Z <001>  1141 MB info    |   filter size                   1.500000
2019-08-12T11:49:24.990807Z <001>  1141 MB info    |   crop window                   (0, 0)-(1,919, 1,079)
2019-08-12T11:49:24.990807Z <001>  1141 MB info    |   denoising mode                off
2019-08-12T11:49:24.990807Z <001>  1141 MB info    |   render stamp                  off
2019-08-12T11:49:24.990807Z <001>  1141 MB info    |   save extra aovs               off
2019-08-12T11:49:24.991217Z <002>  1141 MB info    | --- beginning rendering pass 1 ---
2019-08-12T11:49:25.181753Z <003>  1344 MB info    | rendering, 0.2% done
2019-08-12T11:49:25.182029Z <004>  1344 MB info    | rendering, 0.4% done
2019-08-12T11:49:25.186642Z <005>  1345 MB info    | rendering, 0.6% done
2019-08-12T11:49:25.200894Z <006>  1345 MB info    | rendering, 0.8% done
2019-08-12T11:49:25.201368Z <007>  1346 MB info    | rendering, 1.0% done
2019-08-12T11:49:25.204811Z <008>  1346 MB info    | rendering, 1.2% done
2019-08-12T11:49:25.205593Z <009>  1347 MB info    | rendering, 1.4% done
2019-08-12T11:49:25.209655Z <010>  1347 MB info    | rendering, 1.6% done
2019-08-12T11:49:25.366424Z <003>

I have left it at this point for over 36 hours without progress.

samhodge commented 5 years ago

Operating system is CenOS 7.4 Computer is a HP Z600

(base) [kognat@vxfhost gaffer-0.53.4.0-linux]$ cat /proc/cpuinfo 
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 0
siblings    : 6
core id     : 0
cpu cores   : 6
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5320.12
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 0
siblings    : 6
core id     : 1
cpu cores   : 6
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5320.12
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 0
siblings    : 6
core id     : 2
cpu cores   : 6
apicid      : 4
initial apicid  : 4
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5320.12
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 0
siblings    : 6
core id     : 8
cpu cores   : 6
apicid      : 16
initial apicid  : 16
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5320.12
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 4
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 0
siblings    : 6
core id     : 9
cpu cores   : 6
apicid      : 18
initial apicid  : 18
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5320.12
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 5
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 0
siblings    : 6
core id     : 10
cpu cores   : 6
apicid      : 20
initial apicid  : 20
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5320.12
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 6
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 1
siblings    : 6
core id     : 0
cpu cores   : 6
apicid      : 32
initial apicid  : 32
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5319.75
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 7
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 1
siblings    : 6
core id     : 1
cpu cores   : 6
apicid      : 34
initial apicid  : 34
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5319.75
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 8
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 1
siblings    : 6
core id     : 2
cpu cores   : 6
apicid      : 36
initial apicid  : 36
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5319.75
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 9
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 1
siblings    : 6
core id     : 8
cpu cores   : 6
apicid      : 48
initial apicid  : 48
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5319.75
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 10
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 1
siblings    : 6
core id     : 9
cpu cores   : 6
apicid      : 50
initial apicid  : 50
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5319.75
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor   : 11
vendor_id   : GenuineIntel
cpu family  : 6
model       : 44
model name  : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping    : 2
microcode   : 0x1f
cpu MHz     : 1596.000
cache size  : 12288 KB
physical id : 1
siblings    : 6
core id     : 10
cpu cores   : 6
apicid      : 52
initial apicid  : 52
fpu     : yes
fpu_exception   : yes
cpuid level : 11
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb tpr_shadow vnmi flexpriority ept vpid dtherm ida arat
bogomips    : 5319.75
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:
uname -a
Linux vxfhost 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
(base) [kognat@vxfhost gaffer-0.53.4.0-linux]$ cat /opt/gaffer/gaffer-0.53.4.0-linux/appleseed/include/foundation/core/version.h 

//
// This source file is part of appleseed.
// Visit http://appleseedhq.net/ for additional information and resources.
//
// This software is released under the MIT license.
//
// Copyright (c) 2014-2017 Esteban Tovagliari, The appleseedhq Organization
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
//

#ifndef APPLESEED_FOUNDATION_CORE_VERSION_H
#define APPLESEED_FOUNDATION_CORE_VERSION_H

#define APPLESEED_VERSION_MAJOR     1
#define APPLESEED_VERSION_MINOR     9
#define APPLESEED_VERSION_PATCH     0
#define APPLESEED_VERSION_MATURITY  "beta"

#define APPLESEED_VERSION               \
    APPLESEED_VERSION_MAJOR * 10000 +   \
    APPLESEED_VERSION_MINOR * 100 +     \
    APPLESEED_VERSION_PATCH

#define APPLESEED_VERSION_STRING        \
    "1.9.0-beta"

#endif  // !APPLESEED_FOUNDATION_CORE_VERSION_H
samhodge commented 5 years ago

Seems like there is a little but of TGA reading but it totally doesnt make sense why the previous frame was OK then frame 67 stops.

Attached is a perf record of the process sitting at the 1.6% rendered in the log.

see: http://www.brendangregg.com/perf.html

perf.data.zip

samhodge commented 5 years ago

OK so you can edit the .xml appleseed file and point to the images:

here is the .appleseed archive

http://bit.ly/head-github-samh-castle-frame67

this would be a list of the paths to update

From 3dscan store https://www.3dscanstore.com/blog/Free-3D-Head-Model

(base) [kognat@vxfhost castle_github]$ more castle_github.0067.appleseed | grep Filename
                    <parameter name="Filename" value="string /home/kognat/head/TGA Textures/Head/TGA/Colour_16k.tga" />
                    <parameter name="Filename" value="string /home/kognat/head/TGA Textures/Head/TGA/Normal Map_SubDivision_1.tga" />
                    <parameter name="Filename" value="string /home/kognat/head/TGA Textures/Head/TGA/Translucency.tga" />
                    <parameter name="Filename" value="string /home/kognat/head/TGA Textures/Head/TGA/Spec_16k.tga" />
                    <parameter name="Filename" value="string /home/kognat/head/TGA Textures/Head/TGA/Gloss_8k.tga" />
                    <parameter name="Filename" value="string /home/kognat/head/TGA Textures/Jacket/TGA/Jacket_Colour.tga" />
                    <parameter name="Filename" value="string /home/kognat/head/TGA Textures/Jacket/TGA/Jacket_Spec.tga" />
                    <parameter name="Filename" value="string /home/kognat/gafferbot_comp/plates/plate.0067.png" />

outputs

(base) [kognat@vxfhost castle_github]$ more castle_github.0067.appleseed | grep 0067.exr
            <parameter name="output_filename" value="/home/kognat/head/render/beauty.0067.exr" />
                    <parameter name="output_filename" value="/home/kognat/head/render/pixel_time.0067.exr" />
                    <parameter name="output_filename" value="/home/kognat/head/render/depth.0067.exr" />
                    <parameter name="output_filename" value="/home/kognat/head/render/normal.0067.exr" />
                    <parameter name="output_filename" value="/home/kognat/head/render/emission.0067.exr" />
                    <parameter name="output_filename" value="/home/kognat/head/render/glossy.0067.exr" />
                    <parameter name="output_filename" value="/home/kognat/head/render/diffuse.0067.exr" />

HDRI from http://bit.ly/castle-HDR-kognat

(base) [kognat@vxfhost castle_github]$ more castle_github.0067.appleseed | grep castle.exr
            <parameter name="filename" value="/home/kognat/Downloads/castle.exr" />
samhodge commented 5 years ago

OK I built appleseed from master

tried to render the file

(base) [kognat@vxfhost sandbox]$ ./bin/Ship/appleseed.cli ~/gaffer/projects/default/appleseeds/castle_github/castle_github-edit.0067.appleseed 
2019-08-17T03:07:44.713013Z <001>    28 MB info    | loading project file /home/kognat/gaffer/projects/default/appleseeds/castle_github/castle_github-edit.0067.appleseed...
2019-08-17T03:07:44.717038Z <001>    29 MB info    | scanning /opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/appleseed for plugins...
2019-08-17T03:07:44.717352Z <001>    29 MB info    | scanning /opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/blenderseed for plugins...
2019-08-17T03:07:44.717473Z <001>    29 MB info    | scanning /opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/gaffer for plugins...
2019-08-17T03:07:44.717598Z <001>    29 MB info    | scanning /opt/gaffer/gaffer-0.53.4.0-linux/shaders for plugins...
2019-08-17T03:07:44.772204Z <001>    37 MB info    | loaded mesh file /home/kognat/gaffer/projects/default/appleseeds/castle_github/_geometry/2e0cc81e0d763806397eb32bb7cebbc2.binarymesh (1 object, 60,792 vertices, 117,504 triangles) in 45 ms.
2019-08-17T03:07:44.792321Z <001>    42 MB info    | loaded mesh file /home/kognat/gaffer/projects/default/appleseeds/castle_github/_geometry/7152b1122889f2c097629b93a7a59b3b.binarymesh (1 object, 29,204 vertices, 56,448 triangles) in 20 ms.
2019-08-17T03:07:44.819146Z <001>    51 MB info    | loaded mesh file /home/kognat/gaffer/projects/default/appleseeds/castle_github/_geometry/4b9c1bafe6508fec6502585cc1fb73a6.binarymesh (1 object, 39,932 vertices, 77,184 triangles) in 27 ms.
2019-08-17T03:07:44.827571Z <001>    51 MB info    | loaded mesh file /home/kognat/gaffer/projects/default/appleseeds/castle_github/_geometry/d44f7e6ee94ab3a07c187724e4f98a18.binarymesh (1 object, 12,466 vertices, 24,820 triangles) in 8 ms.
2019-08-17T03:07:45.146333Z <001>   110 MB info    | loaded mesh file /home/kognat/gaffer/projects/default/appleseeds/castle_github/_geometry/3170d380a9abd36eaed4854469211ac4.binarymesh (1 object, 502,664 vertices, 1,003,904 triangles) in 319 ms.
2019-08-17T03:07:45.147012Z <001>   110 MB info    | loaded mesh file /home/kognat/gaffer/projects/default/appleseeds/castle_github/_geometry/201a95c946c2d51ef53d88d92f9219c4.binarymesh (1 object, 441 vertices, 800 triangles) in 0 ms.
2019-08-17T03:07:45.147406Z <001>   110 MB info    | loaded mesh file /home/kognat/gaffer/projects/default/appleseeds/castle_github/_geometry/38533c5e82a2ee73dcdaf4d9c75cf82e.binarymesh (1 object, 441 vertices, 800 triangles) in 0 ms.
2019-08-17T03:07:45.147508Z <001>   110 MB info    | collapsing mesh object "//group/plane.mesh" (441 vertices, 800 triangles, 1 motion segment) to static mesh.
2019-08-17T03:07:45.208628Z <001>   125 MB info    | loaded mesh file /home/kognat/gaffer/projects/default/appleseeds/castle_github/_geometry/cb846937fdfc868770179d70e140639c.binarymesh (1 object, 91,540 vertices, 178,560 triangles) in 59 ms.
2019-08-17T03:07:45.215863Z <001>   125 MB info    | scene bounding box: (-0.888747, -0.208627, -0.964264)-(1.068345, 0.554549, 0.992827).
2019-08-17T03:07:45.215863Z <001>   125 MB info    | scene bounding sphere: center (0.089799, 0.172961, 0.014282), diameter 2.871037.
2019-08-17T03:07:45.245775Z <001>   177 MB info    | migrating project format from revision 25 to revision 26...
2019-08-17T03:07:45.245901Z <001>   177 MB info    | migrating project format from revision 26 to revision 27...
2019-08-17T03:07:45.246004Z <001>   177 MB info    | migrating project format from revision 27 to revision 28...
2019-08-17T03:07:45.246078Z <001>   177 MB info    | migrating project format from revision 28 to revision 29...
2019-08-17T03:07:45.246133Z <001>   177 MB info    | migrating project format from revision 29 to revision 30...
2019-08-17T03:07:45.246210Z <001>   177 MB info    | migrating project format from revision 30 to revision 31...
2019-08-17T03:07:45.246297Z <001>   177 MB info    | successfully loaded project file /home/kognat/gaffer/projects/default/appleseeds/castle_github/castle_github-edit.0067.appleseed in 534 ms (0 error, 0 warning).
2019-08-17T03:07:45.246633Z <001>   177 MB info    | rendering frame...
2019-08-17T03:07:45.255902Z <001>   178 MB info    | opening texture file /home/kognat/Downloads/castle.exr and reading metadata...
2019-08-17T03:07:45.264583Z <001>   179 MB info    | building 5376x2688 importance map for environment edf "/scene//group/light"...
2019-08-17T03:07:48.182725Z <001>   765 MB info    | built importance map for environment edf "/scene//group/light" in 2.9 seconds.
2019-08-17T03:07:48.183857Z <001>   599 MB info    | setting oiio texture cache size to 953.7 MB.
2019-08-17T03:07:48.183957Z <001>   599 MB info    | setting oiio search paths to /opt/gaffer/gaffer-0.53.4.0-linux/shaders:/opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/gaffer:/opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/blenderseed:/opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/appleseed:/home/kognat/gaffer/projects/default/appleseeds/castle_github
2019-08-17T03:07:48.184329Z <001>   600 MB info    | setting osl shader search paths to /opt/gaffer/gaffer-0.53.4.0-linux/shaders:/opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/gaffer:/opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/blenderseed:/opt/gaffer/gaffer-0.53.4.0-linux/appleseed/shaders/appleseed:/home/kognat/gaffer/projects/default/appleseeds/castle_github
2019-08-17T03:07:48.184467Z <001>   600 MB info    | found OSL headers in /home/kognat/dev/appleseed/sandbox/shaders/stdosl.h
/home/esteban.tovagliari/Devel/builds/OpenShadingLanguage/src/liboslexec/llvm_gen.cpp:3228: failed assertion 'Key.typespec().is_string()'
Aborted (core dumped)

I am not sure if the .oso stuff needs to be recompiled from the version in Gaffer. castle_github-edit.0067.appleseed.zip

The .oso files are the ones that come with gaffer 0.53.4

samhodge commented 5 years ago

(base) [kognat@vxfhost sandbox]$ ./bin/Ship/appleseed.cli -ut 2019-08-17T03:16:48.524838Z <001> 42 MB warning | aov "/beauty/indirect_diffuse" cannot be saved to png file; saving it to exr file instead. 2019-08-17T03:16:48.852247Z <001> 42 MB warning | aovs cannot be saved to png files; saving them to exr files instead. 2019-08-17T03:16:50.436988Z <001> 49 MB info | unit testing summary: 2019-08-17T03:16:50.437106Z <001> 49 MB info | suites : 189 executed, 0 failed (0.0%) 2019-08-17T03:16:50.437106Z <001> 49 MB info | cases : 1,570 executed, 0 failed (0.0%) 2019-08-17T03:16:50.437214Z <001> 49 MB info | assertions : 389,645 executed, 0 failed (0.0%)

samhodge commented 5 years ago

(base) [kognat@vxfhost sandbox]$ ./bin/Ship/appleseed.cli -ut 2019-08-17T03:16:48.524838Z <001> 42 MB warning | aov "/beauty/indirect_diffuse" cannot be saved to png file; saving it to exr file instead. 2019-08-17T03:16:48.852247Z <001> 42 MB warning | aovs cannot be saved to png files; saving them to exr files instead. 2019-08-17T03:16:50.436988Z <001> 49 MB info | unit testing summary: 2019-08-17T03:16:50.437106Z <001> 49 MB info | suites : 189 executed, 0 failed (0.0%) 2019-08-17T03:16:50.437106Z <001> 49 MB info | cases : 1,570 executed, 0 failed (0.0%) 2019-08-17T03:16:50.437214Z <001> 49 MB info | assertions : 389,645 executed, 0 failed (0.0%) (base) [kognat@vxfhost sandbox]$ ./bin/Ship/appleseed.cli --version 2019-08-17T03:17:37.488025Z <001> 28 MB info | appleseed.cli, using appleseed.so version 2.1.0-beta, Release configuration 2019-08-17T03:17:37.488025Z <001> 28 MB info | compiled on Aug 17 2019 at 12:07:57 using gcc version 4.8.5 2019-08-17T03:17:37.488025Z <001> 28 MB info | copyright (c) 2010-2013 Francois Beaune, Jupiter Jazz Limited 2019-08-17T03:17:37.488025Z <001> 28 MB info | copyright (c) 2014-2019 The appleseedhq Organization 2019-08-17T03:17:37.488025Z <001> 28 MB info | this software is released under the MIT license (https://opensource.org/licenses/MIT). 2019-08-17T03:17:37.488025Z <001> 28 MB info | visit https://appleseedhq.net/ for additional information and resources. 2019-08-17T03:17:37.488332Z <001> 28 MB info | library features: 2019-08-17T03:17:37.488332Z <001> 28 MB info | Instruction sets SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 2019-08-17T03:17:37.488332Z <001> 28 MB info | Disney material with SeExpr support disabled 2019-08-17T03:17:37.488332Z <001> 28 MB info | Embree enabled 2019-08-17T03:17:37.488332Z <001> 28 MB info | Spectral support enabled 2019-08-17T03:17:37.488332Z <001> 28 MB info | GPU support disabled (base) [kognat@vxfhost sandbox]$ ./bin/Ship/appleseed.cli --system 2019-08-17T03:18:07.328971Z <001> 28 MB info | system information: 2019-08-17T03:18:07.328971Z <001> 28 MB info | architecture x86 64-bit 2019-08-17T03:18:07.328971Z <001> 28 MB info | vendor Intel 2019-08-17T03:18:07.328971Z <001> 28 MB info | logical cores 12 2019-08-17T03:18:07.328971Z <001> 28 MB info | L1 data cache size 32.0 KB, line size 64 bytes 2019-08-17T03:18:07.328971Z <001> 28 MB info | L2 cache size 256.0 KB, line size 64 bytes 2019-08-17T03:18:07.328971Z <001> 28 MB info | L3 cache size 12.0 MB, line size 64 bytes 2019-08-17T03:18:07.328971Z <001> 28 MB info | instruction sets SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 2019-08-17T03:18:07.328971Z <001> 28 MB info | physical memory size 23.5 GB 2019-08-17T03:18:07.328971Z <001> 28 MB info | virtual memory size 35.3 GB 2019-08-17T03:18:07.328971Z <001> 28 MB info | default wallclock timer 1,000,000 Hz 2019-08-17T03:18:07.328971Z <001> 28 MB info | default processor timer 1,000,000,000 Hz (base) [kognat@vxfhost sandbox]$ ./bin/Ship/appleseed.cli --libraries 2019-08-17T03:18:17.814782Z <001> 28 MB info | third party libraries: 2019-08-17T03:18:17.815081Z <001> 28 MB info | BCD v1.1 2019-08-17T03:18:17.815152Z <001> 28 MB info | Boost 1.61.0 2019-08-17T03:18:17.815249Z <001> 28 MB info | Embree 3.5.2 2019-08-17T03:18:17.815311Z <001> 28 MB info | IlmBase 2.3.0 2019-08-17T03:18:17.815373Z <001> 28 MB info | libjpeg-turbo 1.3.1 2019-08-17T03:18:17.815434Z <001> 28 MB info | LibTIFF 4.0.3 2019-08-17T03:18:17.815494Z <001> 28 MB info | LZ4 1.8.3 2019-08-17T03:18:17.815556Z <001> 28 MB info | OpenColorIO 1.1.0 2019-08-17T03:18:17.815624Z <001> 28 MB info | OpenEXR 2.3.0 2019-08-17T03:18:17.815687Z <001> 28 MB info | OpenImageIO 2.0.4 2019-08-17T03:18:17.815748Z <001> 28 MB info | OpenShadingLanguage 1.10.3 2019-08-17T03:18:17.815809Z <001> 28 MB info | SeExpr appleseedhq/SeExpr, qt5-appleseed branch, commit 8a3d1cd 2019-08-17T03:18:17.815872Z <001> 28 MB info | Xerces-C++ 3.2.2 2019-08-17T03:18:17.815935Z <001> 28 MB info | zlib 1.2.7

samhodge commented 5 years ago

@est77 Do you have any ideas?

dictoon commented 5 years ago

Hi @samhodge, thanks for the detailed bug report. Looking it now.

dictoon commented 5 years ago

I reproduced the issue and explained the problem in Discord, but here's the executive summary:

All these factors combined cause catastrophic slowdown when rendering tiles covered by the head (where most if not all textures are used).

A simple solution is to increase the texture store size. I've tried with 8 GB and it works fine, smaller values may also work but I haven't tried.

Another solution (possibly combined with better texture store size settings) is to convert textures to the tx format with the maketx utility provided with appleseed. The tx format is tiled and optimized for performance.