search

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
14.78k stars 430 forks source link

VM/FSM page is lost in image layer #2601

Closed knizhnik closed 4 months ago

knizhnik commented 2 years ago

Steps to reproduce

It is not clear how to reproduce it. We just know final result (at production). There are the following errors:

Oct 10 04:03:04 zenith-1-ps-2.local pageserver[2090969]: 2022-10-10T04:03:04.381482Z ERROR handle_pagerequests{tenant_id=466afeb4c3ec920550ec70403f44fe3a timeline_id=e87aa756960a9f2ee7dfc2360b75f4e5}: error reading relation or page version: could not find data for key 000000067F0000400200000B180200000000 at LSN 1/A6AE8498, for request at LSN 2/131A5908
Oct 10 04:03:04 zenith-1-ps-2.local pageserver[2090969]: Caused by:
Oct 10 04:03:04 zenith-1-ps-2.local pageserver[2090969]:     0: layer traversal: result Missing, cont_lsn 1/A6AE8498, layer: 000000000000000000000000000000000000-000000067F00004002000280000300000000__00000001A6AE8498
Oct 10 04:03:04 zenith-1-ps-2.local pageserver[2090969]:     1: layer traversal: result Continue, cont_lsn 1/A6AE8499, layer: 000000067F000032AC000040040000000000-030000000000000000000000000000000002__00000001A02445E9-00000001A6F3B8D

Key 000000067F0000400200000B180200000000 corresponds to first block of VM page of relation 2840. It is really not present in the image layer:

 ~/zenith/target/release/dump_layerfile 000000000000000000000000000000000000-000000067F00004002000280000300000000__00000001A6AE8498  | fgrep -i 000000067F0000400200000B180
key: 000000067f0000400200000b180000000000 offset 33346589
key: 000000067f0000400200000b180000000001 offset 33354785
key: 000000067f0000400200000b180000000002 offset 33362981
key: 000000067f0000400200000b180000000003 offset 33371177
key: 000000067f0000400200000b180000000004 offset 33379373
key: 000000067f0000400200000b180000000005 offset 33387569
key: 000000067f0000400200000b180000000006 offset 33395765
key: 000000067f0000400200000b180000000007 offset 33403961
key: 000000067f0000400200000b180000000008 offset 33412157
key: 000000067f0000400200000b180000000009 offset 33420353
key: 000000067f0000400200000b18000000000a offset 33428549
key: 000000067f0000400200000b18000000000b offset 33436745
key: 000000067f0000400200000b18000000000c offset 33444941
key: 000000067f0000400200000b18000000000d offset 33453137
key: 000000067f0000400200000b18000000000e offset 33461333
key: 000000067f0000400200000b18000000000f offset 33469529
key: 000000067f0000400200000b180000000010 offset 33477725
key: 000000067f0000400200000b180000000011 offset 33485921
key: 000000067f0000400200000b180000000012 offset 33494117
key: 000000067f0000400200000b180000000013 offset 33502313
key: 000000067f0000400200000b180000000014 offset 33510509
key: 000000067f0000400200000b180000000015 offset 33518705
key: 000000067f0000400200000b180000000016 offset 33526901
key: 000000067f0000400200000b180000000017 offset 33535097
key: 000000067f0000400200000b180000000018 offset 33543293
key: 000000067f0000400200000b180000000019 offset 33551489
key: 000000067f0000400200000b18000000001a offset 33559685
key: 000000067f0000400200000b18000000001b offset 33567881
key: 000000067f0000400200000b18000000001c offset 33576077
key: 000000067f0000400200000b18000000001d offset 33584273
key: 000000067f0000400200000b18000000001e offset 33592469
key: 000000067f0000400200000b18000000001f offset 33600665
key: 000000067f0000400200000b180000000020 offset 33608861
key: 000000067f0000400200000b180000000021 offset 33617057
key: 000000067f0000400200000b180000000022 offset 33625253
key: 000000067f0000400200000b180000000023 offset 33633449
key: 000000067f0000400200000b180000000024 offset 33641645
key: 000000067f0000400200000b180000000025 offset 33649841
key: 000000067f0000400200000b180000000026 offset 33658037
key: 000000067f0000400200000b180000000027 offset 33666233
key: 000000067f0000400200000b180000000028 offset 33674429
key: 000000067f0000400200000b180000000029 offset 33682625
key: 000000067f0000400200000b18000000002a offset 33690821
key: 000000067f0000400200000b18000000002b offset 33699017
key: 000000067f0000400200000b18000000002c offset 33707213
key: 000000067f0000400200000b18000000002d offset 33715409
key: 000000067f0000400200000b18000000002e offset 33723605
key: 000000067f0000400200000b18000000002f offset 33731801
key: 000000067f0000400200000b180000000030 offset 33739997
key: 000000067f0000400200000b180000000031 offset 33748193
key: 000000067f0000400200000b180000000032 offset 33756389
key: 000000067f0000400200000b180000000033 offset 33764585
key: 000000067f0000400200000b180000000034 offset 33772781
key: 000000067f0000400200000b180000000035 offset 33780977
key: 000000067f0000400200000b180000000036 offset 33789173
key: 000000067f0000400200000b180000000037 offset 33797369
key: 000000067f0000400200000b180000000038 offset 33805565
key: 000000067f0000400200000b180000000039 offset 33813761
key: 000000067f0000400200000b18000000003a offset 33821957
key: 000000067f0000400200000b18000000003b offset 33830153
key: 000000067f0000400200000b18000000003c offset 33838349
key: 000000067f0000400200000b18000000003d offset 33846545
key: 000000067f0000400200000b1800ffffffff offset 33854741
key: 000000067f0000400200000b1801ffffffff offset 33854746
key: 000000067f0000400200000b1802ffffffff offset 33854751

Expected result

Actual result

Environment

zenith-1-ps-2 project: withered-king tenant-id: 466afeb4c3ec920550ec70403f44fe3a

Logs, links

Discussion in Slack: https://neondb.slack.com/archives/C03F5SM1N02/p1665362497649899

knizhnik commented 2 years ago

Previous image layer contains this key:

dump_layerfile 000000000000000000000000000000000000-000000067F00004002000280000300000000__0000000193F09060 |  fgrep -i 000000067F0000400200000B180
...
key: 000000067f0000400200000b180100000000 offset 33813766
key: 000000067f0000400200000b180100000001 offset 33821962
key: 000000067f0000400200000b180100000002 offset 33830158
key: 000000067f0000400200000b1801ffffffff offset 33838354
key: 000000067f0000400200000b180200000000 offset 33838359
key: 000000067f0000400200000b1802ffffffff offset 33846555

But intermediate delta layers do not contain this key:

-rw-r--r-- 1 pageserver pageserver  97566720 Oct  8 05:25 000000000000000000000000000000000000-000000067F00004002000280000300000000__00000001A6AE8498
-rw-r--r-- 1 pageserver pageserver 134365184 Oct  8 05:24 000000067F000032AC000040040000000000-030000000000000000000000000000000002__00000001A02445E9-00000001A6F3B8D9
-rw-r--r-- 1 pageserver pageserver  11018240 Oct  8 03:03 000000067F000032AC000040040000000000-030000000000000000000000000000000002__000000019FA3CDE1-00000001A02445E9
-rw-r--r-- 1 pageserver pageserver 241557504 Oct  8 00:50 000000067F000032AC000040040000000000-030000000000000000000000000000000002__00000001938C96B1-000000019FA3CDE1
-rw-r--r-- 1 pageserver pageserver  78143488 Oct  7 22:22 000000067F00004002000280030000000000-030000000000000000000000000000000002__0000000193F09060
-rw-r--r-- 1 pageserver pageserver  97550336 Oct  7 22:22 000000000000000000000000000000000000-000000067F00004002000280000300000000__0000000193F09060
knizhnik commented 2 years ago

I print content of relation sizes (*ffffffff keys). Size of relation itself is not changed and is correct - 0x3e. But size of FSM/VM forks is changed from 3/1 to 0/0. This is why pages were not include in new image layer. The question is who cause truncation of this FSM/VM forks...

knizhnik commented 2 years ago

Looks like I know the answer for my question: we are ot supportig partial truncation of FSM/VM map. and as far as we assume this data to be supplementary, we just completely truncate them:

            // FIXME: 'blkno' stored in the WAL record is the new size of the
            // heap. The formula for calculating the new size of the FSM is
            // pretty complicated (see FreeSpaceMapPrepareTruncateRel() in
            // PostgreSQL), and we should also clear bits in the tail FSM block,
            // and update the upper level FSM pages. None of that has been
            // implemented. What we do instead, is always just truncate the FSM
            // to zero blocks. That's bad for performance, but safe. (The FSM
            // isn't needed for correctness, so we could also leave garbage in
            // it. Seems more tidy to zap it away.)
            if rec.blkno != 0 {
                info!("Partial truncation of FSM is not supported");
            }
            let num_fsm_blocks = 0;
            self.put_rel_truncation(modification, rel, num_fsm_blocks)?;
arssher commented 2 years ago

Interesting. Then compute shouldn't treat absense of FSM page as an error? And similar for compaction.

knizhnik commented 2 years ago

It is one of the possible fixes. Something like it is doe now for size of this forks:

        if (tag.forknum == FSM_FORKNUM || tag.forknum == VISIBILITYMAP_FORKNUM)
            && !self.get_rel_exists(tag, lsn, latest)?
        {
            // FIXME: Postgres sometimes calls smgrcreate() to create
            // FSM, and smgrnblocks() on it immediately afterwards,
            // without extending it.  Tolerate that by claiming that
            // any non-existent FSM fork has size 0.
            return Ok(0);
        }

But I have two concerns:

  1. It is still not quite clear to me why some page of FSM/VM fork is accessed by get_page and page reconstruction. Requesting this page by compute can be explained by local relation cache. But why it is requested during compaction is more obscure.
  2. I am not sure that it correct just to return zero page in this case.
hlinnaka commented 2 years ago

Interesting. Then compute shouldn't treat absense of FSM page as an error? And similar for compaction.

PostgreSQL tolerates missing FSM pages. They are not even WAL-logged (unless wal_log_hints=on). I thought it tolerates missing VM pages too, but now I'm not sure.

knizhnik commented 2 years ago

For VM the only constraint is enforced by Postgres: if page is marked as "all-visible" in VM, then it should also marked as all-visible in heap. But it is allows that page containing all-visible bit in the header, is not parked as all-visible in VM.

knizhnik commented 2 years ago

I still failed to create test reproducing the problem (working on it), but I have implemented correct truncation of VM/FSM forks which should be useful not only for fixing this issue. There is still some "fog" here: truncation is done in one place, filling tail of the last page with zeros - in another (FPI). Also there are multiple checks in original postgres code that relation exists and is larger then specified truncation position. But I hope that we now correctly implement this semantic.

hlinnaka commented 2 years ago

So, where do we stand with this? The compaction is still failing on that project in production:

Oct 19 01:05:04 zenith-1-ps-2.local pageserver[2090969]: 2022-10-19T01:05:04.024220Z  INFO gc_loop{tenant_id=466afeb4c3ec920550ec70403f44fe3a}:gc_timeline{timeline=e87aa756960a9f2ee7dfc2360b75f4e5 cutoff=0/803E400}: Nothing to GC: new_gc_cutoff_lsn 0/803E400, latest_gc_cutoff_lsn 0/803E400
Oct 19 01:05:04 zenith-1-ps-2.local pageserver[2090969]: 2022-10-19T01:05:04.024190Z ERROR compaction_loop{tenant_id=466afeb4c3ec920550ec70403f44fe3a}: Compaction failed, retrying: could not find data for key 000000067F0000400200000B180200000000 at LSN 1/A6AE8498, for request at LSN 3/31892F08:  
layer traversal: result Missing, cont_lsn 1/A6AE8498, layer: 000000000000000000000000000000000000-000000067F00004002000280000300000000__00000001A6AE8498:  
layer traversal: result Continue, cont_lsn 1/A6AE8499, layer: 000000067F000032AC000040040000000000-030000000000000000000000000000000002__00000001A02445E9-00000001A6F3B8D9:  
layer traversal: result Continue, cont_lsn 1/A6F3B8D9, layer: 000000067F000032AC000040040000000000-030000000000000000000000000000000002__00000001A6F3B8D9-00000001A7705D49:  
layer traversal: result Continue, cont_lsn 1/A7705D49, layer: 000000067F000032AC000040040000000000-030000000000000000000000000000000002__00000001A7705D49-00000001AED588D1:  
layer traversal: result Continue, cont_lsn 1/AED588D1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AED588D1-00000001AED59751:  
layer traversal: result Continue, cont_lsn 1/AED59751, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AED59751-00000001AED5AA19:  
layer traversal: result Continue, cont_lsn 1/AED5AA19, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AED5AA19-00000001AED5BB99:  
layer traversal: result Continue, cont_lsn 1/AED5BB99, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AED5BB99-00000001AED88301:  
layer traversal: result Continue, cont_lsn 1/AED88301, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AED88301-00000001AED89781:  
layer traversal: result Continue, cont_lsn 1/AED89781, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AED89781-00000001AF60FD79:  
layer traversal: result Continue, cont_lsn 1/AF60FD79, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF60FD79-00000001AF611811:  
layer traversal: result Continue, cont_lsn 1/AF611811, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF611811-00000001AF6130E9:  
layer traversal: result Continue, cont_lsn 1/AF6130E9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF6130E9-00000001AF616959:  
layer traversal: result Continue, cont_lsn 1/AF616959, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF616959-00000001AF6177D9:  
layer traversal: result Continue, cont_lsn 1/AF6177D9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF6177D9-00000001AF6193E1:  
layer traversal: result Continue, cont_lsn 1/AF6193E1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF6193E1-00000001AF61A579:  
layer traversal: result Continue, cont_lsn 1/AF61A579, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF61A579-00000001AF61C319:  
layer traversal: result Continue, cont_lsn 1/AF61C319, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF61C319-00000001AF61D799:  
layer traversal: result Continue, cont_lsn 1/AF61D799, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF61D799-00000001AF656781:  
layer traversal: result Continue, cont_lsn 1/AF656781, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF656781-00000001AF657F01:  
layer traversal: result Continue, cont_lsn 1/AF657F01, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF657F01-00000001AF659FA9:  
layer traversal: result Continue, cont_lsn 1/AF659FA9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001AF659FA9-00000001B5E2B909:  
layer traversal: result Continue, cont_lsn 1/B5E2B909, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001B5E2B909-00000001C41E9701:  
layer traversal: result Continue, cont_lsn 1/C41E9701, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001C41E9701-00000001D2796CA1:  
layer traversal: result Continue, cont_lsn 1/D2796CA1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001D2796CA1-00000001DDD43679:  
layer traversal: result Continue, cont_lsn 1/DDD43679, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DDD43679-00000001DDDC4F41:  
layer traversal: result Continue, cont_lsn 1/DDDC4F41, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DDDC4F41-00000001DDDC60D9:  
layer traversal: result Continue, cont_lsn 1/DDDC60D9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DDDC60D9-00000001DDDC8791:  
layer traversal: result Continue, cont_lsn 1/DDDC8791, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DDDC8791-00000001DDDCC479:  
layer traversal: result Continue, cont_lsn 1/DDDCC479, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DDDCC479-00000001DDDCDBF9:  
layer traversal: result Continue, cont_lsn 1/DDDCDBF9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DDDCDBF9-00000001DDDD05B1:  
layer traversal: result Continue, cont_lsn 1/DDDD05B1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DDDD05B1-00000001DDDD2049:  
layer traversal: result Continue, cont_lsn 1/DDDD2049, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DDDD2049-00000001DE0816F1:  
layer traversal: result Continue, cont_lsn 1/DE0816F1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DE0816F1-00000001DE5AD4F1:  
layer traversal: result Continue, cont_lsn 1/DE5AD4F1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DE5AD4F1-00000001DE694B71:  
layer traversal: result Continue, cont_lsn 1/DE694B71, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DE694B71-00000001DE695CF1:  
layer traversal: result Continue, cont_lsn 1/DE695CF1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DE695CF1-00000001DE6A4359:  
layer traversal: result Continue, cont_lsn 1/DE6A4359, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DE6A4359-00000001DE6A7079:  
layer traversal: result Continue, cont_lsn 1/DE6A7079, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001DE6A7079-00000001E5984DC1:  
layer traversal: result Continue, cont_lsn 1/E5984DC1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E5984DC1-00000001E5998551:  
layer traversal: result Continue, cont_lsn 1/E5998551, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E5998551-00000001E599B729:  
layer traversal: result Continue, cont_lsn 1/E599B729, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E599B729-00000001E599D1C1:  
layer traversal: result Continue, cont_lsn 1/E599D1C1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E599D1C1-00000001E599ED69:  
layer traversal: result Continue, cont_lsn 1/E599ED69, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E599ED69-00000001E59A0B69:  
layer traversal: result Continue, cont_lsn 1/E59A0B69, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E59A0B69-00000001E59A28C9:  
layer traversal: result Continue, cont_lsn 1/E59A28C9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E59A28C9-00000001E59A5641:  
layer traversal: result Continue, cont_lsn 1/E59A5641, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E59A5641-00000001E59A67D9:  
layer traversal: result Continue, cont_lsn 1/E59A67D9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E59A67D9-00000001E59A8869:  
layer traversal: result Continue, cont_lsn 1/E59A8869, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E59A8869-00000001E59A9CE9:  
layer traversal: result Continue, cont_lsn 1/E59A9CE9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E59A9CE9-00000001E59ABEC9:  
layer traversal: result Continue, cont_lsn 1/E59ABEC9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E59ABEC9-00000001E59AD661:  
layer traversal: result Continue, cont_lsn 1/E59AD661, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E59AD661-00000001E59AF9F9:  
layer traversal: result Continue, cont_lsn 1/E59AF9F9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E59AF9F9-00000001E59B5779:  
layer traversal: result Continue, cont_lsn 1/E59B5779, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E59B5779-00000001E6855771:  
layer traversal: result Continue, cont_lsn 1/E6855771, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001E6855771-00000001ECBF63D9:  
layer traversal: result Continue, cont_lsn 1/ECBF63D9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001ECBF63D9-00000001ED9677F1:  
layer traversal: result Continue, cont_lsn 1/ED9677F1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001ED9677F1-00000001EDA4BDA1:  
layer traversal: result Continue, cont_lsn 1/EDA4BDA1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDA4BDA1-00000001EDA58049:  
layer traversal: result Continue, cont_lsn 1/EDA58049, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDA58049-00000001EDA639D9:  
layer traversal: result Continue, cont_lsn 1/EDA639D9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDA639D9-00000001EDA6FC69:  
layer traversal: result Continue, cont_lsn 1/EDA6FC69, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDA6FC69-00000001EDA7B611:  
layer traversal: result Continue, cont_lsn 1/EDA7B611, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDA7B611-00000001EDA878A1:  
layer traversal: result Continue, cont_lsn 1/EDA878A1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDA878A1-00000001EDA93211:  
layer traversal: result Continue, cont_lsn 1/EDA93211, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDA93211-00000001EDA9F4D9:  
layer traversal: result Continue, cont_lsn 1/EDA9F4D9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDA9F4D9-00000001EDB2B751:  
layer traversal: result Continue, cont_lsn 1/EDB2B751, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDB2B751-00000001EDB379E1:  
layer traversal: result Continue, cont_lsn 1/EDB379E1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDB379E1-00000001EDB43389:  
layer traversal: result Continue, cont_lsn 1/EDB43389, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDB43389-00000001EDB4F619:  
layer traversal: result Continue, cont_lsn 1/EDB4F619, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDB4F619-00000001EDB5DDB9:  
layer traversal: result Continue, cont_lsn 1/EDB5DDB9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDB5DDB9-00000001EDB6A061:  
layer traversal: result Continue, cont_lsn 1/EDB6A061, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDB6A061-00000001EDB759B9:  
layer traversal: result Continue, cont_lsn 1/EDB759B9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDB759B9-00000001EDB81C81:  
layer traversal: result Continue, cont_lsn 1/EDB81C81, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDB81C81-00000001EDB8D5F1:  
layer traversal: result Continue, cont_lsn 1/EDB8D5F1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDB8D5F1-00000001EDB998B9:  
layer traversal: result Continue, cont_lsn 1/EDB998B9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDB998B9-00000001EDBA5261:  
layer traversal: result Continue, cont_lsn 1/EDBA5261, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDBA5261-00000001EDBB14F1:  
layer traversal: result Continue, cont_lsn 1/EDBB14F1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDBB14F1-00000001EDBBCE99:  
layer traversal: result Continue, cont_lsn 1/EDBBCE99, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDBBCE99-00000001EDBC9129:  
layer traversal: result Continue, cont_lsn 1/EDBC9129, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDBC9129-00000001EDBD4A99:  
layer traversal: result Continue, cont_lsn 1/EDBD4A99, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDBD4A99-00000001EDBE0D61:  
layer traversal: result Continue, cont_lsn 1/EDBE0D61, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EDBE0D61-00000001EE61D159:  
layer traversal: result Continue, cont_lsn 1/EE61D159, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE61D159-00000001EE62F971:  
layer traversal: result Continue, cont_lsn 1/EE62F971, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE62F971-00000001EE63B621:  
layer traversal: result Continue, cont_lsn 1/EE63B621, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE63B621-00000001EE647BF1:  
layer traversal: result Continue, cont_lsn 1/EE647BF1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE647BF1-00000001EE6538A1:  
layer traversal: result Continue, cont_lsn 1/EE6538A1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE6538A1-00000001EE65FEA9:  
layer traversal: result Continue, cont_lsn 1/EE65FEA9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE65FEA9-00000001EE66BB59:  
layer traversal: result Continue, cont_lsn 1/EE66BB59, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE66BB59-00000001EE678179:  
layer traversal: result Continue, cont_lsn 1/EE678179, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE678179-00000001EE683E11:  
layer traversal: result Continue, cont_lsn 1/EE683E11, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE683E11-00000001EE6903F9:  
layer traversal: result Continue, cont_lsn 1/EE6903F9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE6903F9-00000001EE69C119:  
layer traversal: result Continue, cont_lsn 1/EE69C119, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE69C119-00000001EE6A86E9:  
layer traversal: result Continue, cont_lsn 1/EE6A86E9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE6A86E9-00000001EE6B4399:  
layer traversal: result Continue, cont_lsn 1/EE6B4399, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001EE6B4399-00000001F53AB291:  
layer traversal: result Continue, cont_lsn 1/F53AB291, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F53AB291-00000001F53CD9A9:  
layer traversal: result Continue, cont_lsn 1/F53CD9A9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F53CD9A9-00000001F53D9FB1:  
layer traversal: result Continue, cont_lsn 1/F53D9FB1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F53D9FB1-00000001F53E5C99:  
layer traversal: result Continue, cont_lsn 1/F53E5C99, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F53E5C99-00000001F53F2281:  
layer traversal: result Continue, cont_lsn 1/F53F2281, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F53F2281-00000001F53FDF51:  
layer traversal: result Continue, cont_lsn 1/F53FDF51, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F53FDF51-00000001F540A539:  
layer traversal: result Continue, cont_lsn 1/F540A539, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F540A539-00000001F54161E9:  
layer traversal: result Continue, cont_lsn 1/F54161E9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F54161E9-00000001F54227F1:  
layer traversal: result Continue, cont_lsn 1/F54227F1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F54227F1-00000001F5664FD9:  
layer traversal: result Continue, cont_lsn 1/F5664FD9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F5664FD9-00000001F61D84F1:  
layer traversal: result Continue, cont_lsn 1/F61D84F1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F61D84F1-00000001F61E41A1:  
layer traversal: result Continue, cont_lsn 1/F61E41A1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F61E41A1-00000001F622D151:  
layer traversal: result Continue, cont_lsn 1/F622D151, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F622D151-00000001F6238E39:  
layer traversal: result Continue, cont_lsn 1/F6238E39, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F6238E39-00000001F6245409:  
layer traversal: result Continue, cont_lsn 1/F6245409, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F6245409-00000001F62510F1:  
layer traversal: result Continue, cont_lsn 1/F62510F1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F62510F1-00000001F625D6C1:  
layer traversal: result Continue, cont_lsn 1/F625D6C1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F625D6C1-00000001F6269371:  
layer traversal: result Continue, cont_lsn 1/F6269371, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F6269371-00000001F6275979:  
layer traversal: result Continue, cont_lsn 1/F6275979, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F6275979-00000001F6281629:  
layer traversal: result Continue, cont_lsn 1/F6281629, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F6281629-00000001F628DBF9:  
layer traversal: result Continue, cont_lsn 1/F628DBF9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F628DBF9-00000001F62998E1:  
layer traversal: result Continue, cont_lsn 1/F62998E1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F62998E1-00000001F62A5EB1:  
layer traversal: result Continue, cont_lsn 1/F62A5EB1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F62A5EB1-00000001F62B1B61:  
layer traversal: result Continue, cont_lsn 1/F62B1B61, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F62B1B61-00000001F62BE181:  
layer traversal: result Continue, cont_lsn 1/F62BE181, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F62BE181-00000001F62C9E19:  
layer traversal: result Continue, cont_lsn 1/F62C9E19, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F62C9E19-00000001F62D6439:  
layer traversal: result Continue, cont_lsn 1/F62D6439, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F62D6439-00000001F62E20E9:  
layer traversal: result Continue, cont_lsn 1/F62E20E9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F62E20E9-00000001F62EE6B9:  
layer traversal: result Continue, cont_lsn 1/F62EE6B9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F62EE6B9-00000001F62FA3A1:  
layer traversal: result Continue, cont_lsn 1/F62FA3A1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F62FA3A1-00000001F6306971:  
layer traversal: result Continue, cont_lsn 1/F6306971, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F6306971-00000001F6312659:  
layer traversal: result Continue, cont_lsn 1/F6312659, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F6312659-00000001F631EC29:  
layer traversal: result Continue, cont_lsn 1/F631EC29, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001F631EC29-00000001FC8D6449:  
layer traversal: result Continue, cont_lsn 1/FC8D6449, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FC8D6449-00000001FE599101:  
layer traversal: result Continue, cont_lsn 1/FE599101, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE599101-00000001FE5A4DE9:  
layer traversal: result Continue, cont_lsn 1/FE5A4DE9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE5A4DE9-00000001FE5B13F1:  
layer traversal: result Continue, cont_lsn 1/FE5B13F1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE5B13F1-00000001FE5BD0A1:  
layer traversal: result Continue, cont_lsn 1/FE5BD0A1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE5BD0A1-00000001FE5C96A9:  
layer traversal: result Continue, cont_lsn 1/FE5C96A9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE5C96A9-00000001FE5D5391:  
layer traversal: result Continue, cont_lsn 1/FE5D5391, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE5D5391-00000001FE5E1961:  
layer traversal: result Continue, cont_lsn 1/FE5E1961, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE5E1961-00000001FE5ED649:  
layer traversal: result Continue, cont_lsn 1/FE5ED649, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE5ED649-00000001FE5F9C19:  
layer traversal: result Continue, cont_lsn 1/FE5F9C19, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE5F9C19-00000001FE6058C9:  
layer traversal: result Continue, cont_lsn 1/FE6058C9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE6058C9-00000001FE611ED1:  
layer traversal: result Continue, cont_lsn 1/FE611ED1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE611ED1-00000001FE61DB81:  
layer traversal: result Continue, cont_lsn 1/FE61DB81, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE61DB81-00000001FE62A169:  
layer traversal: result Continue, cont_lsn 1/FE62A169, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FE62A169-00000001FF0F9481:  
layer traversal: result Continue, cont_lsn 1/FF0F9481, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF0F9481-00000001FF10BBF1:  
layer traversal: result Continue, cont_lsn 1/FF10BBF1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF10BBF1-00000001FF117911:  
layer traversal: result Continue, cont_lsn 1/FF117911, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF117911-00000001FF123EE1:  
layer traversal: result Continue, cont_lsn 1/FF123EE1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF123EE1-00000001FF12FB91:  
layer traversal: result Continue, cont_lsn 1/FF12FB91, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF12FB91-00000001FF13C1B1:  
layer traversal: result Continue, cont_lsn 1/FF13C1B1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF13C1B1-00000001FF147E49:  
layer traversal: result Continue, cont_lsn 1/FF147E49, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF147E49-00000001FF154469:  
layer traversal: result Continue, cont_lsn 1/FF154469, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF154469-00000001FF160119:  
layer traversal: result Continue, cont_lsn 1/FF160119, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF160119-00000001FF16C6E9:  
layer traversal: result Continue, cont_lsn 1/FF16C6E9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF16C6E9-00000001FF1783D1:  
layer traversal: result Continue, cont_lsn 1/FF1783D1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF1783D1-00000001FF1849A1:  
layer traversal: result Continue, cont_lsn 1/FF1849A1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF1849A1-00000001FF190689:  
layer traversal: result Continue, cont_lsn 1/FF190689, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF190689-00000001FF19CC59:  
layer traversal: result Continue, cont_lsn 1/FF19CC59, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF19CC59-00000001FF1A8909:  
layer traversal: result Continue, cont_lsn 1/FF1A8909, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF1A8909-00000001FF1B4A69:  
layer traversal: result Continue, cont_lsn 1/FF1B4A69, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF1B4A69-00000001FF1C0719:  
layer traversal: result Continue, cont_lsn 1/FF1C0719, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF1C0719-00000001FF1CC409:  
layer traversal: result Continue, cont_lsn 1/FF1CC409, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF1CC409-00000001FF1D65E9:  
layer traversal: result Continue, cont_lsn 1/FF1D65E9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF1D65E9-00000001FF1DA689:  
layer traversal: result Continue, cont_lsn 1/FF1DA689, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF1DA689-00000001FF1DBE09:  
layer traversal: result Continue, cont_lsn 1/FF1DBE09, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF1DBE09-00000001FF1DEA81:  
layer traversal: result Continue, cont_lsn 1/FF1DEA81, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF1DEA81-00000001FF1E3311:  
layer traversal: result Continue, cont_lsn 1/FF1E3311, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000001FF1E3311-0000000206140631:  
layer traversal: result Continue, cont_lsn 2/6140631, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000206140631-00000002061531E9:  
layer traversal: result Continue, cont_lsn 2/61531E9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002061531E9-0000000206154081:  
layer traversal: result Continue, cont_lsn 2/6154081, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000206154081-0000000206157181:  
layer traversal: result Continue, cont_lsn 2/6157181, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000206157181-000000020615B611:  
layer traversal: result Continue, cont_lsn 2/615B611, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020615B611-000000020615CAA9:  
layer traversal: result Continue, cont_lsn 2/615CAA9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020615CAA9-000000020615E681:  
layer traversal: result Continue, cont_lsn 2/615E681, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020615E681-000000020615FE01:  
layer traversal: result Continue, cont_lsn 2/615FE01, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020615FE01-00000002069FBC19:  
layer traversal: result Continue, cont_lsn 2/69FBC19, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002069FBC19-00000002069FD6B1:  
layer traversal: result Continue, cont_lsn 2/69FD6B1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002069FD6B1-00000002069FF591:  
layer traversal: result Continue, cont_lsn 2/69FF591, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002069FF591-0000000206A03409:  
layer traversal: result Continue, cont_lsn 2/6A03409, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000206A03409-0000000206A042A1:  
layer traversal: result Continue, cont_lsn 2/6A042A1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000206A042A1-0000000206A0AAF9:  
layer traversal: result Continue, cont_lsn 2/6A0AAF9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000206A0AAF9-0000000208570AB1:  
layer traversal: result Continue, cont_lsn 2/8570AB1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000208570AB1-000000020858F1D1:  
layer traversal: result Continue, cont_lsn 2/858F1D1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020858F1D1-000000020AD57741:  
layer traversal: result Continue, cont_lsn 2/AD57741, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020AD57741-000000020AD5E0B1:  
layer traversal: result Continue, cont_lsn 2/AD5E0B1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020AD5E0B1-000000020AD64A09:  
layer traversal: result Continue, cont_lsn 2/AD64A09, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020AD64A09-000000020AD6B361:  
layer traversal: result Continue, cont_lsn 2/AD6B361, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020AD6B361-000000020AD71CB9:  
layer traversal: result Continue, cont_lsn 2/AD71CB9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020AD71CB9-000000020AD78629:  
layer traversal: result Continue, cont_lsn 2/AD78629, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020AD78629-000000020AD7EF81:  
layer traversal: result Continue, cont_lsn 2/AD7EF81, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020AD7EF81-000000020AD858D9:  
layer traversal: result Continue, cont_lsn 2/AD858D9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020AD858D9-000000020AD8D451:  
layer traversal: result Continue, cont_lsn 2/AD8D451, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020AD8D451-000000020AD93DA9:  
layer traversal: result Continue, cont_lsn 2/AD93DA9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020AD93DA9-000000020AD9A719:  
layer traversal: result Continue, cont_lsn 2/AD9A719, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020AD9A719-000000020ADA1071:  
layer traversal: result Continue, cont_lsn 2/ADA1071, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020ADA1071-000000020ADA79C9:  
layer traversal: result Continue, cont_lsn 2/ADA79C9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020ADA79C9-000000020ADAE339:  
layer traversal: result Continue, cont_lsn 2/ADAE339, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020ADAE339-000000020FDC1AD9:  
layer traversal: result Continue, cont_lsn 2/FDC1AD9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000020FDC1AD9-00000002128136E9:  
layer traversal: result Continue, cont_lsn 2/128136E9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002128136E9-000000021286D141:  
layer traversal: result Continue, cont_lsn 2/1286D141, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021286D141-00000002128784C9:  
layer traversal: result Continue, cont_lsn 2/128784C9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002128784C9-0000000212883831:  
layer traversal: result Continue, cont_lsn 2/12883831, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000212883831-000000021288EBB9:  
layer traversal: result Continue, cont_lsn 2/1288EBB9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021288EBB9-0000000212899EE9:  
layer traversal: result Continue, cont_lsn 2/12899EE9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000212899EE9-00000002128A52A9:  
layer traversal: result Continue, cont_lsn 2/128A52A9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002128A52A9-00000002128B05F1:  
layer traversal: result Continue, cont_lsn 2/128B05F1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002128B05F1-00000002128BB999:  
layer traversal: result Continue, cont_lsn 2/128BB999, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002128BB999-00000002128C6D19:  
layer traversal: result Continue, cont_lsn 2/128C6D19, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002128C6D19-00000002128D20A1:  
layer traversal: result Continue, cont_lsn 2/128D20A1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002128D20A1-00000002128DD409:  
layer traversal: result Continue, cont_lsn 2/128DD409, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002128DD409-00000002128E87C9:  
layer traversal: result Continue, cont_lsn 2/128E87C9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002128E87C9-00000002128F3AF9:  
layer traversal: result Continue, cont_lsn 2/128F3AF9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002128F3AF9-00000002128FEEB9:  
layer traversal: result Continue, cont_lsn 2/128FEEB9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002128FEEB9-000000021290A201:  
layer traversal: result Continue, cont_lsn 2/1290A201, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021290A201-0000000212915571:  
layer traversal: result Continue, cont_lsn 2/12915571, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000212915571-0000000212920929:  
layer traversal: result Continue, cont_lsn 2/12920929, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000212920929-000000021292BC99:  
layer traversal: result Continue, cont_lsn 2/1292BC99, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021292BC99-0000000212937019:  
layer traversal: result Continue, cont_lsn 2/12937019, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000212937019-00000002129423D9:  
layer traversal: result Continue, cont_lsn 2/129423D9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002129423D9-000000021294D709:  
layer traversal: result Continue, cont_lsn 2/1294D709, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021294D709-000000021295A0A9:  
layer traversal: result Continue, cont_lsn 2/1295A0A9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021295A0A9-0000000212966191:  
layer traversal: result Continue, cont_lsn 2/12966191, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000212966191-0000000212972B99:  
layer traversal: result Continue, cont_lsn 2/12972B99, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000212972B99-000000021297EC49:  
layer traversal: result Continue, cont_lsn 2/1297EC49, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021297EC49-0000000213163149:  
layer traversal: result Continue, cont_lsn 2/13163149, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000213163149-000000021316F269:  
layer traversal: result Continue, cont_lsn 2/1316F269, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021316F269-000000021317BC71:  
layer traversal: result Continue, cont_lsn 2/1317BC71, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021317BC71-0000000213187D21:  
layer traversal: result Continue, cont_lsn 2/13187D21, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000213187D21-0000000213194741:  
layer traversal: result Continue, cont_lsn 2/13194741, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000213194741-00000002131A0829:  
layer traversal: result Continue, cont_lsn 2/131A0829, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002131A0829-00000002131AD231:  
layer traversal: result Continue, cont_lsn 2/131AD231, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002131AD231-00000002131B9351:  
layer traversal: result Continue, cont_lsn 2/131B9351, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002131B9351-00000002131C5D21:  
layer traversal: result Continue, cont_lsn 2/131C5D21, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002131C5D21-00000002131D1E09:  
layer traversal: result Continue, cont_lsn 2/131D1E09, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002131D1E09-00000002131DE861:  
layer traversal: result Continue, cont_lsn 2/131DE861, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002131DE861-00000002131EA911:  
layer traversal: result Continue, cont_lsn 2/131EA911, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002131EA911-00000002131F7351:  
layer traversal: result Continue, cont_lsn 2/131F7351, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002131F7351-0000000213203401:  
layer traversal: result Continue, cont_lsn 2/13203401, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000213203401-000000021320FE09:  
layer traversal: result Continue, cont_lsn 2/1320FE09, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021320FE09-000000021321BF29:  
layer traversal: result Continue, cont_lsn 2/1321BF29, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021321BF29-00000002197FC9B9:  
layer traversal: result Continue, cont_lsn 2/197FC9B9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002197FC9B9-000000021B6F7E29:  
layer traversal: result Continue, cont_lsn 2/1B6F7E29, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021B6F7E29-000000021B70EA81:  
layer traversal: result Continue, cont_lsn 2/1B70EA81, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021B70EA81-000000021B72F709:  
layer traversal: result Continue, cont_lsn 2/1B72F709, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021B72F709-000000021B73C0F1:  
layer traversal: result Continue, cont_lsn 2/1B73C0F1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021B73C0F1-000000021B7481D9:  
layer traversal: result Continue, cont_lsn 2/1B7481D9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021B7481D9-000000021B754C19:  
layer traversal: result Continue, cont_lsn 2/1B754C19, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021B754C19-000000021BAAE081:  
layer traversal: result Continue, cont_lsn 2/1BAAE081, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021BAAE081-000000021C1234C1:  
layer traversal: result Continue, cont_lsn 2/1C1234C1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C1234C1-000000021C12F5A9:  
layer traversal: result Continue, cont_lsn 2/1C12F5A9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C12F5A9-000000021C13BFE9:  
layer traversal: result Continue, cont_lsn 2/1C13BFE9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C13BFE9-000000021C148159:  
layer traversal: result Continue, cont_lsn 2/1C148159, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C148159-000000021C154B61:  
layer traversal: result Continue, cont_lsn 2/1C154B61, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C154B61-000000021C160C81:  
layer traversal: result Continue, cont_lsn 2/1C160C81, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C160C81-000000021C16D6C1:  
layer traversal: result Continue, cont_lsn 2/1C16D6C1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C16D6C1-000000021C1797E1:  
layer traversal: result Continue, cont_lsn 2/1C1797E1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C1797E1-000000021C186271:  
layer traversal: result Continue, cont_lsn 2/1C186271, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C186271-000000021C192391:  
layer traversal: result Continue, cont_lsn 2/1C192391, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C192391-000000021C19EDD1:  
layer traversal: result Continue, cont_lsn 2/1C19EDD1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C19EDD1-000000021C1AAEF1:  
layer traversal: result Continue, cont_lsn 2/1C1AAEF1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C1AAEF1-000000021C1B7969:  
layer traversal: result Continue, cont_lsn 2/1C1B7969, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C1B7969-000000021C1C3A89:  
layer traversal: result Continue, cont_lsn 2/1C1C3A89, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C1C3A89-000000021C1D0519:  
layer traversal: result Continue, cont_lsn 2/1C1D0519, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C1D0519-000000021C1DC639:  
layer traversal: result Continue, cont_lsn 2/1C1DC639, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C1DC639-000000021C1E9079:  
layer traversal: result Continue, cont_lsn 2/1C1E9079, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C1E9079-000000021C1F51D1:  
layer traversal: result Continue, cont_lsn 2/1C1F51D1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C1F51D1-000000021C201C11:  
layer traversal: result Continue, cont_lsn 2/1C201C11, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C201C11-000000021C20DD69:  
layer traversal: result Continue, cont_lsn 2/1C20DD69, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C20DD69-000000021C21A7F9:  
layer traversal: result Continue, cont_lsn 2/1C21A7F9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021C21A7F9-000000021DAFD151:  
layer traversal: result Continue, cont_lsn 2/1DAFD151, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000021DAFD151-0000000224241051:  
layer traversal: result Continue, cont_lsn 2/24241051, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000224241051-000000022424D171:  
layer traversal: result Continue, cont_lsn 2/2424D171, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000022424D171-0000000224259BB1:  
layer traversal: result Continue, cont_lsn 2/24259BB1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000224259BB1-0000000224265CD1:  
layer traversal: result Continue, cont_lsn 2/24265CD1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000224265CD1-0000000224272729:  
layer traversal: result Continue, cont_lsn 2/24272729, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000224272729-000000022427E8F9:  
layer traversal: result Continue, cont_lsn 2/2427E8F9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000022427E8F9-000000022428B371:  
layer traversal: result Continue, cont_lsn 2/2428B371, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__000000022428B371-0000000224297491:  
layer traversal: result Continue, cont_lsn 2/24297491, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000224297491-00000002242A3ED1:  
layer traversal: result Continue, cont_lsn 2/242A3ED1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002242A3ED1-00000002242B0041:  
layer traversal: result Continue, cont_lsn 2/242B0041, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002242B0041-00000002242BCAB9:  
layer traversal: result Continue, cont_lsn 2/242BCAB9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002242BCAB9-00000002242C8BD9:  
layer traversal: result Continue, cont_lsn 2/242C8BD9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002242C8BD9-00000002242D5619:  
layer traversal: result Continue, cont_lsn 2/242D5619, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002242D5619-00000002242E1739:  
layer traversal: result Continue, cont_lsn 2/242E1739, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002242E1739-00000002246211C9:  
layer traversal: result Continue, cont_lsn 2/246211C9, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000002246211C9-0000000224C941B1:  
layer traversal: result Continue, cont_lsn 2/24C941B1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000224C941B1-0000000224CA0BF1:  
layer traversal: result Continue, cont_lsn 2/24CA0BF1, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000224CA0BF1-0000000224CF4491:  
layer traversal: result Continue, cont_lsn 2/24CF4491, layer: 000000000000000000000000000000000000-FFFFFFFFFFFFFFF
Oct 19 01:05:06 zenith-1-ps-2.local pageserver[2090969]: 2022-10-19T01:05:06.832059Z  INFO compaction_loop{tenant_id=466afeb4c3ec920550ec70403f44fe3a}:compact_timeline{timeline=e87aa756960a9f2ee7dfc2360b75f4e5}: new image layer /storage/pageserver/data/tenants/466afeb4c3ec920550ec70403f44fe3a/timelines/e87aa756960a9f2ee7dfc2360b75f4

And we have now accumulated quite a lot of delta layers, with no image layers. That is affecting startup time, because basebackup now takes a long time:

Oct 19 01:06:30 zenith-1-ps-2.local pageserver[2090969]: 2022-10-19T01:06:30.390184Z  INFO handle_basebackup_request{tenant_id=466afeb4c3ec920550ec70403f44fe3a timeline_id=e87aa756960a9f2ee7dfc2360b75f4e5 lsn=Some(3/318E2178) prev_lsn=None full_backup=false}: waiting for 3/318E2178
Oct 19 01:06:30 zenith-1-ps-2.local pageserver[2090969]: 2022-10-19T01:06:30.390277Z  INFO handle_basebackup_request{tenant_id=466afeb4c3ec920550ec70403f44fe3a timeline_id=e87aa756960a9f2ee7dfc2360b75f4e5 lsn=Some(3/318E2178) prev_lsn=None full_backup=false}: taking basebackup lsn=3/318E2178, prev_lsn=3/318E2140 (full_backup=false)
Oct 19 01:06:43 zenith-1-ps-2.local pageserver[2090969]: 2022-10-19T01:06:43.758398Z  INFO handle_basebackup_request{tenant_id=466afeb4c3ec920550ec70403f44fe3a timeline_id=e87aa756960a9f2ee7dfc2360b75f4e5 lsn=Some(3/318E2178) prev_lsn=None full_backup=false lsn="3/318E2178"}: timeline.pg_version 14
Oct 19 01:06:44 zenith-1-ps-2.local pageserver[2090969]: 2022-10-19T01:06:44.125201Z  INFO handle_basebackup_request{tenant_id=466afeb4c3ec920550ec70403f44fe3a timeline_id=e87aa756960a9f2ee7dfc2360b75f4e5 lsn=Some(3/318E2178) prev_lsn=None full_backup=false lsn="3/318E2178"}: basebackup complete
knizhnik commented 2 years ago

There is PR #2609 waiting for review...

arssher commented 2 years ago

I'm ready to provide env for replaying WAL for this project on staging so we can test the fix, which appears to be https://github.com/neondatabase/neon/pull/2609

knizhnik commented 2 years ago

But #2609 will not fix already incorrectly truncated VM/FSM relations. So after applying the patch the problem with this project will not be solved automatically.

arssher commented 2 years ago

Yes, we will need to replay on prod too.

hlinnaka commented 2 years ago

But #2609 will not fix already incorrectly truncated VM/FSM relations. So after applying the patch the problem with this project will not be solved automatically.

If I understand correctly, it also doesn't explain why the key went missing in the storage. Even if the compute generates an incorrect sequence of WAL records, it should not lead to an internal inconsistency like that in the layer files.

knizhnik commented 2 years ago

There is no incorrect sequence of WAL records. It is incorrect interpretation of correct sequence of WAL records by page server: it truncates FSM/VM map till zero rather than til requested position.

hlinnaka commented 2 years ago

There is no incorrect sequence of WAL records. It is incorrect interpretation of correct sequence of WAL records by page server: it truncates FSM/VM map till zero rather than til requested position.

Ok, but even then, the pageserver should correctly truncate the FSM/VM maps to zero. It shouldn't cause an internal inconsistency between the relation size stored in the layer files, and the pages that exist.

hlinnaka commented 2 years ago

The top of the trace above:

Oct 19 01:05:04 zenith-1-ps-2.local pageserver[2090969]: 2022-10-19T01:05:04.024190Z ERROR compaction_loop{tenant_id=466afeb4c3ec920550ec70403f44fe3a}: Compaction failed, retrying: could not find data for key 000000067F0000400200000B180200000000 at LSN 1/A6AE8498, for request at LSN 3/31892F08:  layer traversal: result Missing, cont_lsn 1/A6AE8498, layer: 000000000000000000000000000000000000-000000067F00004002000280000300000000__00000001A6AE8498:  
layer traversal: result Continue, cont_lsn 1/A6AE8499, layer: 000000067F000032AC000040040000000000-030000000000000000000000000000000002__00000001A02445E9-00000001A6F3B8D9:

I looked at those layer files. the image layer (000000000000000000000000000000000000-000000067F00004002000280000300000000__00000001A6AE8498) indeed doesn't have that page, and the relation size key says that the size is 0:

key: 000000067f0000400200000b1800ffffffff offset 33854741 value: 3e000000
key: 000000067f0000400200000b1801ffffffff offset 33854746 value: 00000000
key: 000000067f0000400200000b1802ffffffff offset 33854751 value: 00000000    <- VM fork has size zero

The delta layer (000000067F000032AC000040040000000000-030000000000000000000000000000000002__00000001A02445E9-00000001A6F3B8D9) contains a bunch of updates to the VM page:

  key 000000067F0000400200000B180200000000 at 1/A6147950:  rec 8273 bytes will_init: true will_init: true, HEAP2 VISIBLE
  key 000000067F0000400200000B180200000000 at 1/A61479D0:  rec 76 bytes will_init: false will_init: false, HEAP2 VISIBLE
  [lots more HEAP2_VISIBLE records with will_init: false] 
  key 000000067F0000400200000B180200000000 at 1/A61496A0:  rec 76 bytes will_init: false will_init: false, HEAP2 VISIBLE
  key 000000067F0000400200000B180200000000 at 1/A6149718:  rec 76 bytes will_init: false will_init: false, HEAP2 VISIBLE
  key 000000067F0000400200000B180200000000 at 1/A6149790:  rec 76 bytes will_init: false will_init: false, HEAP2 VISIBLE
  key 000000067F0000400200000B180200000000 at 1/A6DEDF28:  rec 76 bytes will_init: false will_init: false, HEAP2 VISIBLE
  key 000000067F0000400200000B180200000000 at 1/A6DEDF68:  rec 76 bytes will_init: false will_init: false, HEAP2 VISIBLE
  key 000000067F0000400200000B180200000000 at 1/A6DEDFA8:  rec 76 bytes will_init: false will_init: false, HEAP2 VISIBLE
  key 000000067F0000400200000B180200000000 at 1/A6DEF358:  rec 76 bytes will_init: false will_init: false, HEAP2 VISIBLE
  [lots more HEAP2_VISIBLE records with will_init: false] 
  key 000000067F0000400200000B180200000000 at 1/A6DEF268:  rec 76 bytes will_init: false will_init: false, HEAP2 VISIBLE
  key 000000067F0000400200000B180200000000 at 1/A6DEF2E0:  rec 76 bytes will_init: false will_init: false, HEAP2 VISIBLE
  key 000000067F0000400200000B180200000000 at 1/A6DEF358:  rec 76 bytes will_init: false will_init: false, HEAP2 VISIBLE
  key 000000067F0000400200000B1802FFFFFFFF at 1/A61498B0:  img 4 bytes: 00000000
  key 000000067F0000400200000B1802FFFFFFFF at 1/A6DEDF28:  img 4 bytes: 01000000

This looks fishy. The size-entries don't match with the page-entries. 

1/A6147950: will-init record for the page, but no update to size, so size is still 0
1/A61479D0 - 1/A6149790:  incremental updates to the VM page, size is still 0
1/A61498B0: size is updated to 0 (Why? It was already zero)
1/A6DEDF28: update size to 1.
1/A6DEF268 - 1/A6DEF358: more incremental updates the the page

Mysteries:

Also,why does the compaction fail to reconstruct the VM page? ISTM that it should find the will_init=true record at 1/A6147950, and stop the search there.

hlinnaka commented 2 years ago

Also,why does the compaction fail to reconstruct the VM page? ISTM that it should find the will_init=true record at 1/A6147950, and stop the search there.

Oh, I see now that the image layer is in the middle of the LSN range of the delta layer, not before it as I though.

I think I understand this finally. This is indeed caused by the issue with partial truncation that https://github.com/neondatabase/neon/pull/2609 addresses.

Postgres creates a stream of WAL record for the VM page. First, a record with will_init=true, and then a stream of incremental updates. The relation is truncated, which clears some bits in the VM page, and more incremental records follow.

However, in the pageserver we truncate away the VM page completely. But that makes the subsequent incremental WAL records unusable: applying an incremental WAL record requires a before-image, but because we truncated away the page in the pageserver, the before-image cannot be found.

hlinnaka commented 2 years ago

Actions:

Could we make the image creation more robust for errors like that? The safest action is to error out, but I wonder if we could safely try something else too? For example, if we fail to reconstruct an FSM / VM page, perhaps log an error, and record the page as all-zeros in the image layer. For FSM/VM, that should be OK, you won't lose data, and it would unstuck the compaction/image layer creation. Or we if there are overlapping image and delta layers like in this case, and we cannot find the page in the image layer, we could try to reconstruct it using the delta layer instead.

knizhnik commented 2 years ago

It sounds like good way to hide the problem. Yes, if will be logged, but we are not able to inspect all logs for all errors. Such "workaround" can cause DB to return wrong results. Which may be even worse than crash.

knizhnik commented 2 years ago

I wonder what we are going to do with exited projects suffering from this bug in production? Should we really try to create commit soe workaround (like creation of zero page if root page image is not found?

hlinnaka commented 2 years ago

Such "workaround" can cause DB to return wrong results. Which may be even worse than crash.

If we only do it for the VM and FSM pages, it should not lead to wrong results.

knizhnik commented 2 years ago

Such "workaround" can cause DB to return wrong results. Which may be even worse than crash.

If we only do it for the VM and FSM pages, it should not lead to wrong results.

Yes, But do you expect problems only with VM/FSM pages? Yes, this particular bug is caused by truncation of this forks. But it is not quite clear for me - if we expect some more problems with page reconstruction, why they be relate only with this forks?

hlinnaka commented 2 years ago

Such "workaround" can cause DB to return wrong results. Which may be even worse than crash.

If we only do it for the VM and FSM pages, it should not lead to wrong results.

Yes, But do you expect problems only with VM/FSM pages? Yes, this particular bug is caused by truncation of this forks. But it is not quite clear for me - if we expect some more problems with page reconstruction, why they be relate only with this forks?

This bug was about VM and FSM specifically. Sure, we could have different bugs that affect the main fork, and then it wouldn't help. That's OK, we're no worse off for the main fork than without it.

I do actually suspect that bugs like this are more likely with the VM and FSM than with the main fork. The WAL-logging around the FSM and VM is pretty complicated in PostgreSQL. We knowingly violate the usual "WAL-before-data" rule on some updates, and in the past, there have been bugs because of that in PostgreSQL too.

arssher commented 2 years ago

FWIW I coped this project to staging under timeline_id b76b8481fd6f200ab7c6a73139f77eea, and it doesn't seem to trigger compaction error after rolling all WAL: https://observer.zenith.tech/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Loki%22,%7B%22refId%22:%22A%22,%22expr%22:%22%7Bhostname%20%3D~%5C%22.*ps-4.*%5C%22%7D%20%7C%3D%20%5C%22b76b8481fd6f200ab7c6a73139f77eea%5C%22%22%7D%5D

hlinnaka commented 4 months ago

The bug was fixed, the accumulation of temp files on error was fixed, and the production tenant was cleaned up, nothing left to do here.