redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.49k stars 580 forks source link

Offline log viewer (OLV): add strict mode to OLV to fail on incomplete parsing #9943

Open rystsov opened 1 year ago

rystsov commented 1 year ago
### Tasks
- [x] hardening parsing of basic batches of data partition: configuration & raft_data
- [ ] adding support of archival_metadata batches of data partition
- [x] hardening tx batches of data partition: fence, abort & commit
- [x] hardening parsing of controller parsing
- [x] hardening parsing of id_allocator log
- [x] hardening parsing of kvstore log & snapshot
- [ ] hardening parsing of tx coordinator log
- [ ] hardening parsing of consumer offsets
- [ ] hook into a subset of ducktape test generating the above batches

JIRA Link: CORE-1257

rystsov commented 1 year ago

https://github.com/rystsov/redpanda/tree/OLV

rystsov commented 1 year ago

found this issue https://github.com/redpanda-data/redpanda/issues/10214

rystsov commented 1 year ago

Labels

(-) isn't supported (+) read supported (*) write supported

Structure

log type
  batch type
    variations within a batch
(-) data partition
    (*) raft_configuration
    (+) raft_data
        (*) user data 
        (+) tx control marker
    (*) checkpoint
    (+) tx_fence
    (-) archival_metadata
        (-) truncate_cmd
        (-) update_start_offset_cmd
        (-) cleanup_metadata_cmd
        (-) mark_clean_cmd

(+) controller
    (+) cluster_bootstrap_cmd
        (+) bootstrap_cluster_cmd_type
    (+) cluster_config_cmd
        (+) cluster_config_delta_cmd_type
        (+) cluster_config_status_cmd_type
    (+) node_management_cmd
        (+) decommission_node_cmd_type
        (+) recommission_node_cmd_type
        (+) finish_reallocations_cmd_type
        (+) maintenance_mode_cmd_type
        (+) register_node_uuid_cmd_type
        (+) add_node_cmd_type
        (+) update_node_cmd_type
        (+) remove_node_cmd_type
    (+) topic_management_cmd
        (+) create_topic_cmd_type = 0
        (+) delete_topic_cmd_type = 1
        (+) move_partition_replicas_cmd_type = 2
        (+) finish_moving_partition_replicas_cmd_type = 3
        (+) update_topic_properties_cmd_type = 4
        (+) create_partition_cmd_type = 5
        (+) create_non_replicable_topic_cmd_type = 6
        (+) cancel_moving_partition_replicas_cmd_type = 7
        (+) move_topic_replicas_cmd_type = 8
        (+) revert_cancel_partition_move_cmd_type = 9
    (*) raft_configuration
    (*) checkpoint
    (+) feature_update
        (+) feature_update_cmd_type
        (+) feature_update_license_update_cmd_type
    (+) user_management_cmd
        (+) create_user_cmd_type
        (+) delete_user_cmd_type
        (+) update_user_cmd_type
    (+) acl_management_cmd
        (+) create_acls_cmd_type
        (+) delete_acls_cmd_type
    (+) data_policy_management_cmd
        (+) create_data_policy_cmd_type
        (+) delete_data_policy_cmd_type

(+) kvstore
    (+) kvstore
        (+) consensus
            (+) voted_for
            (+) config_map
            (+) config_latest_known_offset
            (+) last_applied_offset
            (+) unique_local_id
            (+) config_next_cfg_idx
        (+) controller
            (+) feature_table
            (+) configuration_invariants
            (+) cluster_members
            (+) cluster_uuid
            (+) node_uuid
        (+) storage
            (+) clean_segment
            (+) start_offset
        (+) offset_translator
            (+) highest_known_offset
            (+) offsets_map
    (+) snapshot

(+) id_allocator
    (*) raft_configuration
    (*) checkpoint
    (+) id_allocator
        (+) allocation_cmd
        (+) prepare_truncation_cmd
        (+) execute_truncation_cmd
        (+) state_cmd

(-) tx coordinator
    (*) raft_configuration
    (*) checkpoint
    (-) tm_update

(-) consumer groups
    (*) raft_configuration
    (*) checkpoint
    (-) raft_data
        (-) offset_commit
        (-) group_metadata
        (-) noop
    (-) group_prepare_tx
    (-) group_commit_tx
    (-) group_abort_tx
    (-) tx_fence
    (-) version_fence