search

bloomberg / bde

Basic Development Environment - a set of foundational C++ libraries used at Bloomberg.
Apache License 2.0
1.68k stars 318 forks source link

Test failure in bdls_filesystemutil.t #283

Closed greghogan closed 1 year ago

greghogan commented 2 years ago

bde 3.110.0.3 bde-tools 3.110.0.0

My Linux filesystem only supports 32-bit timestamps (max-year 2038) so the year 2200 tests are off by 59150 days ~= 162 years.

It is unclear to me why TestUtil_UnixImpUtil::getMinFileTime() returns either a DateTime of bsl::numeric_limits<time_t>::max() seconds or maxDatetime(9999, 12, 31, 23, 59, 59, 999, 999) seemingly without checking that the filesystem supports 64-bit timestamps (which is not guaranteed by having 64-bit off_t).

$ ctest -V -R 'bdls_filesystemutil\.t'
UpdateCTestConfiguration  from :/tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/build/DartConfiguration.tcl
Test project /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 404
    Start 404: bdls_filesystemutil.t

404: Test command: /gnu/store/65i3nhcwmz0p8rqbg48gaavyky4g4hwk-python-3.9.9/bin/python3.9 "/gnu/store/van5j8ddlvsrwqjf9j7jvw4c722jl43c-bloomberg-bde-tools-3.110.0.0/bin/bde_runtest.py" "/tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/build/bdls_filesystemutil.t"
404: Test timeout computed to be: 10000000
404: [14:52:08] TEST START
404: [14:52:08] CASE  2: SUCCESS
404: [14:52:08] CASE  1: SUCCESS
404: [14:52:08] CASE  4: SUCCESS
404: [14:52:08] CASE  5: SUCCESS
404: [14:52:08] CASE  6: SUCCESS
404: [14:52:08] CASE  7: SUCCESS
404: [14:52:08] CASE  8: SUCCESS
404: [14:52:08] CASE  9: SUCCESS
404: [14:52:08] CASE 10: SUCCESS
404: [14:52:08] CASE 11: SUCCESS
404: [14:52:08] CASE  3: SUCCESS
404: [14:52:08] CASE 12: SUCCESS
404: [14:52:08] CASE 13: SUCCESS
404: [14:52:08] CASE 15: SUCCESS
404: [14:52:08] CASE 16: SUCCESS
404: [14:52:09] CASE 14: SUCCESS
404: [14:52:09] CASE 18: SUCCESS
404: [14:52:09] CASE 19: SUCCESS
404: [14:52:09] CASE 17: SUCCESS
404: [14:52:09] CASE 20: SUCCESS
404: [14:52:09] CASE 21: SUCCESS
404: [14:52:09] CASE 22: SUCCESS
404: [14:52:09] CASE 23: SUCCESS
404: [14:52:09] CASE 24: SUCCESS
404: [14:52:09] CASE 25: SUCCESS
404: [14:52:09] CASE 27: SUCCESS
404: [14:52:09] CASE 28: SUCCESS
404: [14:52:11] CASE 26: FAILURE (rc 10)
404: TEST /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp CASE 26
404: LINE: 3436 MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_20:45:53.000000
404: Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
404: LINE: 3437 MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_20:45:53.000001
404: Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
404: LINE: 3438 MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_20:45:53.001000
404: Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
404: LINE: 3439 MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_20:45:54.000000
404: Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
404: LINE: 3440 MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_20:46:53.000000
404: Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
404: LINE: 3441 MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_21:45:53.000000
404: Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
404: LINE: 3442 MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59151_20:45:53.000000
404: Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
404: LINE: 3443 MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59181_20:45:53.000000
404: Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
404: LINE: 3444 MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_22:45:52.999999
404: Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
404: LINE: 3445 MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59515_20:45:52.999999
404: Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
404: Error, non-zero test status = 10.
404: 
1/1 Test #404: bdls_filesystemutil.t ............***Failed    3.12 sec
[14:52:08] TEST START
[14:52:08] CASE  2: SUCCESS
[14:52:08] CASE  1: SUCCESS
[14:52:08] CASE  4: SUCCESS
[14:52:08] CASE  5: SUCCESS
[14:52:08] CASE  6: SUCCESS
[14:52:08] CASE  7: SUCCESS
[14:52:08] CASE  8: SUCCESS
[14:52:08] CASE  9: SUCCESS
[14:52:08] CASE 10: SUCCESS
[14:52:08] CASE 11: SUCCESS
[14:52:08] CASE  3: SUCCESS
[14:52:08] CASE 12: SUCCESS
[14:52:08] CASE 13: SUCCESS
[14:52:08] CASE 15: SUCCESS
[14:52:08] CASE 16: SUCCESS
[14:52:09] CASE 14: SUCCESS
[14:52:09] CASE 18: SUCCESS
[14:52:09] CASE 19: SUCCESS
[14:52:09] CASE 17: SUCCESS
[14:52:09] CASE 20: SUCCESS
[14:52:09] CASE 21: SUCCESS
[14:52:09] CASE 22: SUCCESS
[14:52:09] CASE 23: SUCCESS
[14:52:09] CASE 24: SUCCESS
[14:52:09] CASE 25: SUCCESS
[14:52:09] CASE 27: SUCCESS
[14:52:09] CASE 28: SUCCESS
[14:52:11] CASE 26: FAILURE (rc 10)
TEST /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp CASE 26
LINE: 3436  MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_20:45:53.000000
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
LINE: 3437  MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_20:45:53.000001
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
LINE: 3438  MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_20:45:53.001000
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
LINE: 3439  MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_20:45:54.000000
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
LINE: 3440  MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_20:46:53.000000
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
LINE: 3441  MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_21:45:53.000000
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
LINE: 3442  MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59151_20:45:53.000000
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
LINE: 3443  MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59181_20:45:53.000000
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
LINE: 3444  MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59150_22:45:52.999999
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
LINE: 3445  MOD_TIME_PRECISION: +0_00:00:00.000001  modTimeSkew: +59515_20:45:52.999999
Error /tmp/guix-build-bloomberg-bde-3.110.0.3.drv-0/source/groups/bdl/bdls/bdls_filesystemutil.t.cpp(3510): MOD_TIME_PRECISION > modTimeSkew    (failed)
Error, non-zero test status = 10.

0% tests passed, 1 tests failed out of 1

Label Time Summary:
all                      =   3.12 sec*proc (1 test)
all.t                    =   3.12 sec*proc (1 test)
bdl                      =   3.12 sec*proc (1 test)
bdl.t                    =   3.12 sec*proc (1 test)
bdls                     =   3.12 sec*proc (1 test)
bdls.t                   =   3.12 sec*proc (1 test)
bdls_filesystemutil      =   3.12 sec*proc (1 test)
bdls_filesystemutil.t    =   3.12 sec*proc (1 test)

Total Test time (real) =   3.21 sec

The following tests FAILED:
    404 - bdls_filesystemutil.t (Failed)
Errors while running CTest
mversche commented 2 years ago

I assume we may not have a test-environment with a filesystem with 32 bit time stamps. Could you elaborate on your environment?

greghogan commented 2 years ago

This is compiling on an Amazon Linux 2 EC2 instance with the default xfs filesystem. Running Linux 5.10 but big timestamps are not enabled.

$ mount | grep xfs
/dev/nvme0n1p1 on / type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

$ xfs_info / | grep bigtime

$ uname -r
5.10.130-118.517.amzn2.x86_64

$ sudo /bin/touch -d 22000101 /file && /bin/stat /file && sudo rm /file
  File: ‘/file’
  Size: 0           Blocks: 0          IO Block: 4096   regular empty file
Device: 10301h/66305d   Inode: 2572269     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2038-01-19 03:14:07.000000000 +0000
Modify: 2038-01-19 03:14:07.000000000 +0000
Change: 2022-11-08 18:39:16.840670282 +0000
 Birth: -
mversche commented 2 years ago

https://github.com/bloomberg/bde/blob/main/groups/bdl/bdls/bdls_filesystemutil.t.cpp#L3458 appears meant to handle systems which do not provide 32 bits for their time stamps. I would assume though that the logic being performed here makes compile-time assumption (based on the build environment) about the range of available offsets. I would assume this may be a runtime property based on the file location.

mversche commented 2 years ago

internal tracking 170731132

MungoG commented 2 years ago

Thank you Greg for pointing this out. We are setting up an environment which will enable us to reproduce this (and ensure any fix is suitably tested prior to release), and will let you know how this progresses.

mversche commented 1 year ago

The fix for this has been merged. Should be available once the main branch syncs.

Commit id: 1925c93e7bf8bca1ba4cade05b09cbff07561fd7