[EPIC]: Logs new schema

[nityanandagohain]

• [x] Otel collector changes: https://github.com/SigNoz/signoz-otel-collector/pull/348
  • [x] Write to new logs_v2 table and logs_v2_resource_bucket table
  • [x] Use maps for attributes and resource attributes, and added token_bf index
  • [x] use number instead of int64 and float64
  • [x] use ngram index for body
  • [ ] Update tags to use number instead of int64 and float64, breaking change
• [x] Query service changes: https://github.com/SigNoz/signoz/pull/5528
  • [x] Body search case insensitive and uses ngram
  • [x] even though tags are int64 and float64, they are converted to number.
  • [x] Any different orderby other than timestamp should use the new table.
  • [x] Any different orderby (resource) should be added as a where clause in resource filter. (OR) (since coverd by groupBy)
  • [x] Any different groupby (resource) should be added as a where clause in resource filter. (OR)
  • [x] Use proper indexes in resource filter
  • [x] Proper indexes for attributes/resource attributes for the table ( case insensitive)
  • [ ] Number support in tag attribute API's
• [x] Frontend Changes

  • [ ] Support for number type

    Decisions to be taken

• [ ] How should we handle group by, as of now if there are two fields in group by , we add exists filter with AND while filtering for it in resources, we can use OR here, if we don't use then no filtering will be done. ref:- https://github.com/SigNoz/signoz/pull/5420

• [x] As of now the otel collector writes to two table, the logs_v2 and logs. logs is the old table with order by timestamp, id. Should we keep it the same or create a materialized view for it ?

  We have moved on from logs and will be just using logs_v2

• [x] Do we even require materialized columns ?

  Yes only for group by

• [x] Should filtering on attributes be case independent? decision = yes.

[nityanandagohain]

Few changes

• [x] Since number type on tag_attrbutes will require frontend changes, keeping it as float64 and then in qs we are mapping it to number.
• [x] Add lower case indexing to the resource fingerprint labels.
• [x] Check indexing on attributes_number Only bloom filter index on numbers is working.

  
  Also tried checking out the issue with minmax index for numbers, only bloom filter is working and not the minmax with numbers in maps.
  CREATE TABLE test_table
  (
  id UInt64,
  valueInt Map(String, Int64),
  valueFloat Map(String, Float64),
  onlyVal Float64,
  index valueint_key_index mapKeys(valueInt) TYPE bloom_filter GRANULARITY 4,
  index valuefloat_key_index mapKeys(valueFloat) TYPE bloom_filter GRANULARITY 4,
  INDEX valueint_idx mapValues(valueInt) TYPE minmax GRANULARITY 1,
  INDEX valueint_idx_b mapValues(valueInt) TYPE bloom_filter GRANULARITY 1,
  INDEX valueFloat_idx mapValues(valueFloat) TYPE minmax GRANULARITY 1,
  INDEX valueFloat_idx_b mapValues(valueFloat) TYPE bloom_filter GRANULARITY 1,
   INDEX onlyV onlyVal TYPE minmax GRANULARITY 1,
  INDEX onlyV_b onlyVal TYPE bloom_filter GRANULARITY 1,
  ) ENGINE=MergeTree ORDER BY id;

INSERT INTO test_table SELECT number as id, map(toString(number), number), map(toString(number), number), number FROM numbers(1000000);

--- not working on maps explain indexes=1 select * from test_table where valueFloat['1000'] > 999 limit 10;

explain indexes=1 select * from test_table where valueFloat['1000'] = 1000.0 limit 10;

explain indexes=1 select * from test_table where valueInt['1000'] > 999 limit 10;

explain indexes=1 select * from test_table where valueInt['1000'] = 1000 limit 10;

-- working on raw column explain indexes=1 select * from test_table where onlyVal = 1000.0 limit 10;

explain indexes=1 select * from test_table where onlyVal > 1000.0 limit 10; Also when > , < is used index on the key is also not applied, on when = is used index on the key is applied.

[nityanandagohain]

Decision w.r.t index

• Go ahead with case sensitive for attributes
• Remove minmax index and only keep bloom filter.

Create issues on clickhouse.

cc @srikanthccv

[nityanandagohain]

• [x] remove resource filtering from main table.
• [x] resource attribute filtering should be case independent
• [x] Support for indexing in IN for resource attributes
• [x] check index support for in and create issue for frontend changes to remove from logs detail. In doesn't work with string values.

[nityanandagohain]

https://github.com/SigNoz/signoz/issues/5614

[nityanandagohain]

This left in new schema

• [x] Scope change to map in query service
• [ ] FE change to use = instead of in https://github.com/SigNoz/signoz/issues/5597
• [ ] Minmax Index on numbers don't work (clickhouse limitation)
• [ ] Attribute search is case sensitive, while resources and body search is case insensitive. ( clickhouse limitation)
• [ ] Migration plan for old users
  • [ ] Users who are still using _ attributes
  • [ ] Users who are not using any _ attributes or the normal ones
  • [ ] Users who have a very high TTL set-up with tiered storage
  • [ ] Users who have materialized columns needs to be migrated

• [ ] Default materialized column if any required? (not a blocker or much linked with this issue)

  cc @ankitnayan @raj-k-singh please add if you think anything else. (apart from JSON search which will be addressed separately)