Failure to initial-sync due to malformed events?

[ara4n]

I still can't get my @matthew:matrix.org account to sync, which is a shame as my Riot is completely hosed and i'd love to switch to nheko whilst debugging the Riot.

I get a bunch of JSON exceptions which I assume are non-fatal due to typos in events:

[json.exception.out_of_range.403] key 'join_rule' not found
{
  "content": {
    "join_rules": "invite"
  },
  "event_id": "$144605034793982mSovJ:matrix.org",
  "origin_server_ts": 1446050347494,
  "sender": "@matthew:matrix.org",
  "state_key": "",
  "type": "m.room.join_rules",
  "unsigned": {
    "age": 73288673369
  }
}

...but it looks like it eventually fails with:

   565.084: [warning] - Initial sync error: @dan:52.35.80.194: the domain name is not valid
   565.084: [warning] - Retrying initial sync

...which is a bit harsh, as @dan:52.35.80.194 is a valid mxid.

[mujx]

Both of those errors are related to the parsing done in matrix-structs. I don't know why the first event was even accepted by synapse because the spec agrees with the current implementation.

About the second, probably the regex here is wrong.

[mujx]

FYI the sync might still fail because nheko will stop after a few failures. The PR https://github.com/mujx/nheko/pull/234 will help, although it's not finished.

[ara4n]

@mujx the various invalid events are probably from years ago before synapse was doing better checks. given the existence of E2E, the server simply can't be trusted to validate events sadly and the client needs to be resilient to malformed ones.

Fwiw, the full list it barfed on is...

[json.exception.out_of_range.403] key 'join_rule' not found
{
  "content": {
    "join_rules": "invite"
  },
  "event_id": "$144605034793982mSovJ:matrix.org",
  "origin_server_ts": 1446050347494,
  "sender": "@matthew:matrix.org",
  "state_key": "",
  "type": "m.room.join_rules",
  "unsigned": {
    "age": 73288673369
  }
}
[json.exception.out_of_range.403] key 'join_rule' not found
{
  "content": {},
  "event_id": "$1499350088673374SehuR:matrix.org",
  "origin_server_ts": 1499350088368,
  "sender": "@matthew:matrix.org",
  "state_key": "",
  "type": "m.room.join_rules",
  "unsigned": {
    "age": 19988932495,
    "prev_content": {
      "join_rule": "invite"
    },
    "prev_sender": "@matthew:matrix.org",
    "replaces_state": "$1499350069673081bOkiq:matrix.org"
  }
}
[json.exception.out_of_range.403] key 'history_visibility' not found
{
  "content": {},
  "event_id": "$1499350088673375hSqUF:matrix.org",
  "origin_server_ts": 1499350088371,
  "sender": "@matthew:matrix.org",
  "state_key": "",
  "type": "m.room.history_visibility",
  "unsigned": {
    "age": 19988932492,
    "prev_content": {
      "history_visibility": "shared"
    },
    "prev_sender": "@matthew:matrix.org",
    "replaces_state": "$1499350069673084fZWWO:matrix.org"
  }
}
[json.exception.out_of_range.403] key 'guest_access' not found
{
  "content": {},
  "event_id": "$1499350088673376HdjsP:matrix.org",
  "origin_server_ts": 1499350088373,
  "sender": "@matthew:matrix.org",
  "state_key": "",
  "type": "m.room.guest_access",
  "unsigned": {
    "age": 19988932490,
    "prev_content": {
      "guest_access": "can_join"
    },
    "prev_sender": "@matthew:matrix.org",
    "replaces_state": "$1499350069673089XIqnU:matrix.org"
  }
}
[json.exception.out_of_range.403] key 'name' not found
{
  "content": {
    "name": "mxisd"
  },
  "event_id": "$14906234641748zZtbq:kamax.io",
  "origin_server_ts": 1490623464125,
  "sender": "@max:kamax.io",
  "state_key": "",
  "type": "m.room.name",
  "unsigned": {
    "age": 28715556738,
    "prev_content": {},
    "prev_sender": "@max:kamax.io",
    "replaces_state": "$148691445421YQYdS:kamax.io"
  }
}
[json.exception.out_of_range.403] key 'alias' not found
{
  "content": {},
  "event_id": "$151576001274vqfkY:lant.uk",
  "origin_server_ts": 1515760012664,
  "sender": "@tom:lant.uk",
  "state_key": "",
  "type": "m.room.canonical_alias",
  "unsigned": {
    "age": 3579008199,
    "prev_content": {
      "alias": "#cam:lant.uk"
    },
    "prev_sender": "@tom:lant.uk",
    "replaces_state": "$151499954197meroK:lant.uk"
  }
}

[mujx]

Got it. Likely, as you pointed out in the first comment, those malformed events aren't fatal and they are simply ignored.

[unclechu]

@ara4n Using mujx/nheko#234 PR I successfully synced my account with 150+ rooms, so it works, you could also try (it took about half of an hour or more).