search

PostHog / charts-clickhouse

Helm chart for deploying PostHog with ClickHouse on your K8s infrastructure
MIT License
56 stars 74 forks source link

Unable get trends of Unique Users. #485

Open yog27ray opened 2 years ago

yog27ray commented 2 years ago

While creating trends with unique users I am getting errors in api response. But when I change it from "unique users" to "Total users" everything is working fine. I even tried doing a fresh deployment but still facing the same issue.

Error in API

URL: https://abc.xyz.com/api/projects/2/insights/trend/?insight=TRENDS&events=%5B%7B%22id%22%3A%22%24pageview%22%2C%22math%22%3A%22dau%22%2C%22type%22%3A%22events%22%2C%22order%22%3A0%7D%5D&interval=week&actions=%5B%5D&new_entity=%5B%5D&display=ActionsLineGraph&properties=%5B%5D

Response: {"type":"server_error","code":"error","detail":"A server error occurred.","attr":null}

values.yml

cloud: "gcp"
ingress:
  hostname: abc.xyz.com
  gcp:
    ip_name: "posthog"
  nginx:
    enabled: true

https://user-images.githubusercontent.com/9247177/178695838-55e635a9-3682-48d7-bb1a-a79b68e49c35.mov

cert-manager:
  enabled: true

# Use larger storage for stateful services
clickhouse:
  enabled: true
  layout:
    shardsCount: 3
    replicasCount: 2
#  nodeSelector:
#    diskType: ssd
#    nodeType: fast
  persistence:
    enabled: true
    size: 100Gi

postgresql:
  enabled: true

https://user-images.githubusercontent.com/9247177/178697346-5f1d7cca-a0b4-42af-86d4-1f51815ac255.mp4

  persistence:
    enabled: true
    size: 60Gi
#  master:
#    nodeSelector:
#      diskType: ssd
#      nodeType: fast
  resources:
    requests:
      cpu: 3000m
      memory: 4Gi
    limits:
      cpu: 3000m
      memory: 4Gi

kafka:
  enabled: true
#  nodeSelector:
#    diskType: ssd
#    nodeType: fast
  persistence:
    enabled: true
    size: 50Gi
  logRetentionBytes: _22_000_000_000
  resources:
    requests:
      cpu: 2000m
      memory: 2Gi
    limits:
      cpu: 2000m
      memory: 2Gi

#redis:
#  master:
#    nodeSelector:
#      diskType: ssd
#      nodeType: fast

#zookeeper:
#  nodeSelector:
#    diskType: ssd
#    nodeType: fast

# Enable horizontal pod autoscaling for stateless services
events:
  hpa:
    enabled: true
    cputhreshold: 60
    minpods: 1
    maxpods: 10
  resources:  
    requests:
      cpu: 1500m
      memory: 2Gi
    limits:
      cpu: 1500m
      memory: 2Gi

pgbouncer:
  hpa:
    enabled: true
    cputhreshold: 60
    minpods: 1
    maxpods: 10
  resources:
    requests:
      cpu: 500m
      memory: 100Mi
    limits:
      cpu: 500m
      memory: 100Mi

plugins:
  hpa:
    enabled: true
    cputhreshold: 60
    minpods: 1
    maxpods: 10
  resources:
    requests:
      cpu: 1500m
      memory: 2Gi
    limits:
      cpu: 1500m
      memory: 2Gi
web:
  hpa:
    enabled: true
    cputhreshold: 60
    minpods: 1
    maxpods: 10
  resources:
    requests:
      cpu: 1000m
      memory: 2Gi
    limits:
      cpu: 1000m
      memory: 2Gi

worker:
  hpa:
    enabled: true
    cputhreshold: 60
    minpods: 1
    maxpods: 10
  resources:                                    
    requests:
      cpu: 1000m
      memory: 1Gi
    limits:
      cpu: 1000m
      memory: 1Gi

# Email
email:
  host: smtp.gmail.com
  port: 587
  user: xxxx@xxx
  password: xxxxx

https://user-images.githubusercontent.com/9247177/178697426-3a152510-5b5c-454c-a1e8-328f64178c6d.mp4

guidoiaquinti commented 2 years ago

👋 Hi! Can you please check the logs from the web service?

yog27ray commented 2 years ago

Hi below is the log from web service.

2022-07-14T03:00:50.044127Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767648958&ver=1.24.0'> request_id=92223c24bb6ee59f7ee4930cf140e63e user_agent=Mozilla/5.0 (Linux; Android 11; RMX3197 Build/RP1A.200720.011; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/87.0.4280.141 Mobile Safari/537.36
2022-07-14T03:00:50.046726Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767648981&ver=1.24.0'> request_id=6e4ea6d1c626c2cf739845cf6b0615c0 user_agent=Mozilla/5.0 (Linux; Android 11; RMX3197 Build/RP1A.200720.011; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/87.0.4280.141 Mobile Safari/537.36
2022-07-14T03:00:50.047289Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767648970&ver=1.24.0'> request_id=1ee21b2c5e26371da69eaaff20b137e3 user_agent=Mozilla/5.0 (Linux; Android 11; RMX3197 Build/RP1A.200720.011; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/87.0.4280.141 Mobile Safari/537.36
2022-07-14T03:00:50.050582Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767648976&ver=1.24.0'> request_id=e7968516a4c44d408bacdd71101e96c2 user_agent=Mozilla/5.0 (Linux; Android 11; RMX3197 Build/RP1A.200720.011; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/87.0.4280.141 Mobile Safari/537.36
2022-07-14T03:00:50.056560Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767648981&ver=1.24.0'> request_id=6e4ea6d1c626c2cf739845cf6b0615c0 user_id=None
2022-07-14T03:00:50.058520Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767648958&ver=1.24.0'> request_id=92223c24bb6ee59f7ee4930cf140e63e user_id=None
2022-07-14T03:00:50.060979Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767648976&ver=1.24.0'> request_id=e7968516a4c44d408bacdd71101e96c2 user_id=None
2022-07-14T03:00:50.065168Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767648970&ver=1.24.0'> request_id=1ee21b2c5e26371da69eaaff20b137e3 user_id=None
2022-07-14T03:00:50.501828Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.1.1 request=<WSGIRequest: GET '/_livez'> request_id=2f1d1a41-6fdd-4fa4-b22a-557ff4d6c627 user_agent=kube-probe/1.23
2022-07-14T03:00:50.502826Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 ip=10.240.1.1 request=<WSGIRequest: GET '/_livez'> request_id=2f1d1a41-6fdd-4fa4-b22a-557ff4d6c627
2022-07-14T03:00:51.333964Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.0.130 request=<WSGIRequest: GET '/api/projects/2/insights/1/?refresh=true&from_dashboard=1'> request_id=e730f276196ebe6dca97bb511aae1826 user_agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36
2022-07-14T03:00:51.527120Z [error    ] Code: 60.
DB::Exception: Received from chi-posthog-posthog-0-0:9000. DB::Exception: Table default.person_distinct_id2 doesn't exist. Stack trace:

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) @ 0xb3ddb64 in /usr/bin/clickhouse
2. void std::__1::__optional_storage_base<DB::Exception, false>::__construct<int const&, char const (&) [23], std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int const&, char const (&) [23], std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) @ 0x1552e94f in /usr/bin/clickhouse
3. DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::__1::shared_ptr<DB::Context const>, std::__1::optional<DB::Exception>*) const @ 0x1551ca61 in /usr/bin/clickhouse
4. DB::DatabaseCatalog::getTable(DB::StorageID const&, std::__1::shared_ptr<DB::Context const>) const @ 0x15524c8b in /usr/bin/clickhouse
5. DB::JoinedTables::getLeftTableStorage() @ 0x15b10b3c in /usr/bin/clickhouse
6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::unordered_map<DB::PreparedSetKey, std::__1::shared_ptr<DB::Set>, DB::PreparedSetKey::Hash, std::__1::equal_to<DB::PreparedSetKey>, std::__1::allocator<std::__1::pair<DB::PreparedSetKey const, std::__1::shared_ptr<DB::Set> > > >) @ 0x15a89326 in /usr/bin/clickhouse
7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15a88594 in /usr/bin/clickhouse
8. DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::__1::shared_ptr<DB::IAST> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae520a in /usr/bin/clickhouse
9. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae3650 in /usr/bin/clickhouse
10. DB::InterpreterSelectWithUnionQuery::getSampleBlock(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, bool) @ 0x15ae60c0 in /usr/bin/clickhouse
11. DB::getDatabaseAndTablesWithColumns(std::__1::vector<DB::ASTTableExpression const*, std::__1::allocator<DB::ASTTableExpression const*> > const&, std::__1::shared_ptr<DB::Context const>, bool, bool) @ 0x15d3c01c in /usr/bin/clickhouse
12. DB::JoinedTables::resolveTables() @ 0x15b10d9f in /usr/bin/clickhouse
13. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::unordered_map<DB::PreparedSetKey, std::__1::shared_ptr<DB::Set>, DB::PreparedSetKey::Hash, std::__1::equal_to<DB::PreparedSetKey>, std::__1::allocator<std::__1::pair<DB::PreparedSetKey const, std::__1::shared_ptr<DB::Set> > > >) @ 0x15a891c9 in /usr/bin/clickhouse
14. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15a88594 in /usr/bin/clickhouse
15. DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::__1::shared_ptr<DB::IAST> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae520a in /usr/bin/clickhouse
16. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae3650 in /usr/bin/clickhouse
17. DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x15a2ecb0 in /usr/bin/clickhouse
18. ? @ 0x15d27821 in /usr/bin/clickhouse
19. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x15d255f5 in /usr/bin/clickhouse
20. DB::TCPHandler::runImpl() @ 0x168b0a3a in /usr/bin/clickhouse
21. DB::TCPHandler::run() @ 0x168c0399 in /usr/bin/clickhouse
22. Poco::Net::TCPServerConnection::start() @ 0x19b801ef in /usr/bin/clickhouse
23. Poco::Net::TCPServerDispatcher::run() @ 0x19b82641 in /usr/bin/clickhouse
24. Poco::PooledThread::run() @ 0x19d3f609 in /usr/bin/clickhouse
25. Poco::ThreadImpl::runnableEntry(void*) @ 0x19d3c960 in /usr/bin/clickhouse
26. ? @ 0x7fd2a989e609 in ?
27. __clone @ 0x7fd2a97c3163 in ?
: While executing Remote. Stack trace:

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::readException(DB::ReadBuffer&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) @ 0xb3cba3a in /usr/bin/clickhouse
2. DB::Connection::receiveException() const @ 0x165aeb81 in /usr/bin/clickhouse
3. DB::Connection::receivePacket() @ 0x165b5b2b in /usr/bin/clickhouse
4. DB::PacketReceiver::Routine::operator()(boost::context::fiber&&) @ 0x165d4085 in /usr/bin/clickhouse
5. void boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, FiberStack&, DB::PacketReceiver::Routine> >(boost::context::detail::transfer_t) @ 0x165d3ece in /usr/bin/clickhouse
 [posthog.exceptions] ip=10.240.0.130 path=/api/projects/2/insights/1/ request_id=e730f276196ebe6dca97bb511aae1826
Traceback (most recent call last):
  File "/home/posthog/code/posthog/client.py", line 179, in sync_execute
    result = client.execute(
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 267, in execute
    rv = self.process_ordinary_query(
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 453, in process_ordinary_query
    return self.receive_result(with_column_types=with_column_types,
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 117, in receive_result
    return result.get_result()
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/result.py", line 50, in get_result
    for packet in self.packet_generator:
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 133, in packet_generator
    packet = self.receive_packet()
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 150, in receive_packet
    raise packet.exception
clickhouse_driver.errors.ServerException: Code: 60.
DB::Exception: Received from chi-posthog-posthog-0-0:9000. DB::Exception: Table default.person_distinct_id2 doesn't exist. Stack trace:

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) @ 0xb3ddb64 in /usr/bin/clickhouse
2. void std::__1::__optional_storage_base<DB::Exception, false>::__construct<int const&, char const (&) [23], std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int const&, char const (&) [23], std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) @ 0x1552e94f in /usr/bin/clickhouse
3. DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::__1::shared_ptr<DB::Context const>, std::__1::optional<DB::Exception>*) const @ 0x1551ca61 in /usr/bin/clickhouse
4. DB::DatabaseCatalog::getTable(DB::StorageID const&, std::__1::shared_ptr<DB::Context const>) const @ 0x15524c8b in /usr/bin/clickhouse
5. DB::JoinedTables::getLeftTableStorage() @ 0x15b10b3c in /usr/bin/clickhouse
6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::unordered_map<DB::PreparedSetKey, std::__1::shared_ptr<DB::Set>, DB::PreparedSetKey::Hash, std::__1::equal_to<DB::PreparedSetKey>, std::__1::allocator<std::__1::pair<DB::PreparedSetKey const, std::__1::shared_ptr<DB::Set> > > >) @ 0x15a89326 in /usr/bin/clickhouse
7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15a88594 in /usr/bin/clickhouse
8. DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::__1::shared_ptr<DB::IAST> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae520a in /usr/bin/clickhouse
9. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae3650 in /usr/bin/clickhouse
10. DB::InterpreterSelectWithUnionQuery::getSampleBlock(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, bool) @ 0x15ae60c0 in /usr/bin/clickhouse
11. DB::getDatabaseAndTablesWithColumns(std::__1::vector<DB::ASTTableExpression const*, std::__1::allocator<DB::ASTTableExpression const*> > const&, std::__1::shared_ptr<DB::Context const>, bool, bool) @ 0x15d3c01c in /usr/bin/clickhouse
12. DB::JoinedTables::resolveTables() @ 0x15b10d9f in /usr/bin/clickhouse
13. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::unordered_map<DB::PreparedSetKey, std::__1::shared_ptr<DB::Set>, DB::PreparedSetKey::Hash, std::__1::equal_to<DB::PreparedSetKey>, std::__1::allocator<std::__1::pair<DB::PreparedSetKey const, std::__1::shared_ptr<DB::Set> > > >) @ 0x15a891c9 in /usr/bin/clickhouse
14. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15a88594 in /usr/bin/clickhouse
15. DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::__1::shared_ptr<DB::IAST> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae520a in /usr/bin/clickhouse
16. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae3650 in /usr/bin/clickhouse
17. DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x15a2ecb0 in /usr/bin/clickhouse
18. ? @ 0x15d27821 in /usr/bin/clickhouse
19. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x15d255f5 in /usr/bin/clickhouse
20. DB::TCPHandler::runImpl() @ 0x168b0a3a in /usr/bin/clickhouse
21. DB::TCPHandler::run() @ 0x168c0399 in /usr/bin/clickhouse
22. Poco::Net::TCPServerConnection::start() @ 0x19b801ef in /usr/bin/clickhouse
23. Poco::Net::TCPServerDispatcher::run() @ 0x19b82641 in /usr/bin/clickhouse
24. Poco::PooledThread::run() @ 0x19d3f609 in /usr/bin/clickhouse
25. Poco::ThreadImpl::runnableEntry(void*) @ 0x19d3c960 in /usr/bin/clickhouse
26. ? @ 0x7fd2a989e609 in ?
27. __clone @ 0x7fd2a97c3163 in ?
: While executing Remote. Stack trace:

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::readException(DB::ReadBuffer&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) @ 0xb3cba3a in /usr/bin/clickhouse
2. DB::Connection::receiveException() const @ 0x165aeb81 in /usr/bin/clickhouse
3. DB::Connection::receivePacket() @ 0x165b5b2b in /usr/bin/clickhouse
4. DB::PacketReceiver::Routine::operator()(boost::context::fiber&&) @ 0x165d4085 in /usr/bin/clickhouse
5. void boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, FiberStack&, DB::PacketReceiver::Routine> >(boost::context::detail::transfer_t) @ 0x165d3ece in /usr/bin/clickhouse

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/posthog/code/posthog/api/insight.py", line 499, in retrieve
    serialized_data = self.get_serializer(instance, context=serializer_context).data
  File "/usr/local/lib/python3.8/site-packages/rest_framework/serializers.py", line 548, in data
    ret = super().data
  File "/usr/local/lib/python3.8/site-packages/rest_framework/serializers.py", line 246, in data
    self._data = self.to_representation(self.instance)
  File "/home/posthog/code/posthog/api/insight.py", line 377, in to_representation
    representation = super().to_representation(instance)
  File "/home/posthog/code/posthog/api/insight.py", line 142, in to_representation
    representation = super().to_representation(instance)
  File "/home/posthog/code/posthog/api/tagged_item.py", line 52, in to_representation
    ret = super(TaggedItemSerializerMixin, self).to_representation(obj)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/serializers.py", line 515, in to_representation
    ret[field.field_name] = field.to_representation(attribute)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/fields.py", line 1870, in to_representation
    return method(value)
  File "/home/posthog/code/posthog/api/insight.py", line 319, in get_result
    return update_insight_cache(insight, dashboard)
  File "/home/posthog/code/posthog/tasks/update_cache.py", line 114, in update_insight_cache
    result = update_cache_item(cache_key, cache_type, payload)
  File "/home/posthog/code/posthog/tasks/update_cache.py", line 65, in update_cache_item
    insight_result = _update_cache_for_queryset(cache_type, filter, key, team, insights_queryset)
  File "/home/posthog/code/posthog/tasks/update_cache.py", line 101, in _update_cache_for_queryset
    raise e
  File "/home/posthog/code/posthog/tasks/update_cache.py", line 93, in _update_cache_for_queryset
    result = _calculate_by_filter(filter, key, team, cache_type)
  File "/home/posthog/code/posthog/tasks/update_cache.py", line 187, in _calculate_by_filter
    result = insight_class().run(filter, team)
  File "/home/posthog/code/posthog/queries/trends/trends.py", line 202, in run
    result.extend(handle_compare(filter, self._run_query, team, entity=entity))
  File "/home/posthog/code/posthog/queries/base.py", line 51, in handle_compare
    trend_entity = func(filter=filter, team=team, **kwargs)
  File "/home/posthog/code/posthog/queries/trends/trends.py", line 125, in _run_query
    result = sync_execute(sql, params)
  File "/home/posthog/code/posthog/client.py", line 188, in sync_execute
    raise err
posthog.errors.CHQueryErrorUnknownTable: Code: 60.
DB::Exception: Received from chi-posthog-posthog-0-0:9000. DB::Exception: Table default.person_distinct_id2 doesn't exist. Stack trace:

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) @ 0xb3ddb64 in /usr/bin/clickhouse
2. void std::__1::__optional_storage_base<DB::Exception, false>::__construct<int const&, char const (&) [23], std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int const&, char const (&) [23], std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) @ 0x1552e94f in /usr/bin/clickhouse
3. DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::__1::shared_ptr<DB::Context const>, std::__1::optional<DB::Exception>*) const @ 0x1551ca61 in /usr/bin/clickhouse
4. DB::DatabaseCatalog::getTable(DB::StorageID const&, std::__1::shared_ptr<DB::Context const>) const @ 0x15524c8b in /usr/bin/clickhouse
5. DB::JoinedTables::getLeftTableStorage() @ 0x15b10b3c in /usr/bin/clickhouse
6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::unordered_map<DB::PreparedSetKey, std::__1::shared_ptr<DB::Set>, DB::PreparedSetKey::Hash, std::__1::equal_to<DB::PreparedSetKey>, std::__1::allocator<std::__1::pair<DB::PreparedSetKey const, std::__1::shared_ptr<DB::Set> > > >) @ 0x15a89326 in /usr/bin/clickhouse
7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15a88594 in /usr/bin/clickhouse
8. DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::__1::shared_ptr<DB::IAST> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae520a in /usr/bin/clickhouse
9. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae3650 in /usr/bin/clickhouse
10. DB::InterpreterSelectWithUnionQuery::getSampleBlock(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, bool) @ 0x15ae60c0 in /usr/bin/clickhouse
11. DB::getDatabaseAndTablesWithColumns(std::__1::vector<DB::ASTTableExpression const*, std::__1::allocator<DB::ASTTableExpression const*> > const&, std::__1::shared_ptr<DB::Context const>, bool, bool) @ 0x15d3c01c in /usr/bin/clickhouse
12. DB::JoinedTables::resolveTables() @ 0x15b10d9f in /usr/bin/clickhouse
13. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::unordered_map<DB::PreparedSetKey, std::__1::shared_ptr<DB::Set>, DB::PreparedSetKey::Hash, std::__1::equal_to<DB::PreparedSetKey>, std::__1::allocator<std::__1::pair<DB::PreparedSetKey const, std::__1::shared_ptr<DB::Set> > > >) @ 0x15a891c9 in /usr/bin/clickhouse
14. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15a88594 in /usr/bin/clickhouse
15. DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::__1::shared_ptr<DB::IAST> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae520a in /usr/bin/clickhouse
16. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15ae3650 in /usr/bin/clickhouse
17. DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x15a2ecb0 in /usr/bin/clickhouse
18. ? @ 0x15d27821 in /usr/bin/clickhouse
19. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x15d255f5 in /usr/bin/clickhouse
20. DB::TCPHandler::runImpl() @ 0x168b0a3a in /usr/bin/clickhouse
21. DB::TCPHandler::run() @ 0x168c0399 in /usr/bin/clickhouse
22. Poco::Net::TCPServerConnection::start() @ 0x19b801ef in /usr/bin/clickhouse
23. Poco::Net::TCPServerDispatcher::run() @ 0x19b82641 in /usr/bin/clickhouse
24. Poco::PooledThread::run() @ 0x19d3f609 in /usr/bin/clickhouse
25. Poco::ThreadImpl::runnableEntry(void*) @ 0x19d3c960 in /usr/bin/clickhouse
26. ? @ 0x7fd2a989e609 in ?
27. __clone @ 0x7fd2a97c3163 in ?
: While executing Remote. Stack trace:

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb37173a in /usr/bin/clickhouse
1. DB::readException(DB::ReadBuffer&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) @ 0xb3cba3a in /usr/bin/clickhouse
2. DB::Connection::receiveException() const @ 0x165aeb81 in /usr/bin/clickhouse
3. DB::Connection::receivePacket() @ 0x165b5b2b in /usr/bin/clickhouse
4. DB::PacketReceiver::Routine::operator()(boost::context::fiber&&) @ 0x165d4085 in /usr/bin/clickhouse
5. void boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, FiberStack&, DB::PacketReceiver::Routine> >(boost::context::detail::transfer_t) @ 0x165d3ece in /usr/bin/clickhouse

2022-07-14T03:00:51.530915Z [info     ] request_finished               [django_structlog.middlewares.request] code=500 ip=10.240.0.130 request=<WSGIRequest: GET '/api/projects/2/insights/1/?refresh=true&from_dashboard=1'> request_id=e730f276196ebe6dca97bb511aae1826 user_id=1
Internal Server Error: /api/projects/2/insights/1/
Internal Server Error: /api/projects/2/insights/1/
2022-07-14T03:00:52.899649Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767652001&ver=1.24.0'> request_id=e572b63121724033b445bc9b0c2810c0 user_agent=Mozilla/5.0 (Linux; Android 10; RMX2030 Build/QKQ1.200209.002; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/89.0.4389.105 Mobile Safari/537.36
2022-07-14T03:00:52.906401Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767652001&ver=1.24.0'> request_id=e572b63121724033b445bc9b0c2810c0 user_id=None
2022-07-14T03:00:53.275339Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767650513&ver=1.24.0'> request_id=eff57bfe8b6b77a575e132c8610aeffe user_agent=Mozilla/5.0 (Linux; Android 8.1.0; CPH1901 Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/70.0.3538.110 Mobile Safari/537.36
2022-07-14T03:00:53.275565Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767650543&ver=1.24.0'> request_id=8afd3f106b8b6cb4fd9ffdd9a4439f23 user_agent=Mozilla/5.0 (Linux; Android 8.1.0; CPH1901 Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/70.0.3538.110 Mobile Safari/537.36
2022-07-14T03:00:53.278365Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767650524&ver=1.24.0'> request_id=3da98c15be8c2b486d30d7851fb005fb user_agent=Mozilla/5.0 (Linux; Android 8.1.0; CPH1901 Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/70.0.3538.110 Mobile Safari/537.36
2022-07-14T03:00:53.276701Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767650533&ver=1.24.0'> request_id=67d1f46246c89f32cffc7bdb95bd0cea user_agent=Mozilla/5.0 (Linux; Android 8.1.0; CPH1901 Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/70.0.3538.110 Mobile Safari/537.36
2022-07-14T03:00:53.285361Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767650513&ver=1.24.0'> request_id=eff57bfe8b6b77a575e132c8610aeffe user_id=None
2022-07-14T03:00:53.287540Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767650533&ver=1.24.0'> request_id=67d1f46246c89f32cffc7bdb95bd0cea user_id=None
2022-07-14T03:00:53.288128Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767650543&ver=1.24.0'> request_id=8afd3f106b8b6cb4fd9ffdd9a4439f23 user_id=None
2022-07-14T03:00:53.290702Z [info     ] request_finished               [django_structlog.middlewares.request] code=200 ip=10.240.0.130 request=<WSGIRequest: POST '/engage/?ip=1&_=1657767650524&ver=1.24.0'> request_id=3da98c15be8c2b486d30d7851fb005fb user_id=None
2022-07-14T03:00:55.502751Z [info     ] request_started                [django_structlog.middlewares.request] ip=10.240.1.1 request=<WSGIRequest: GET '/_livez'> request_id=1f2b034d-0341-450c-9565-9f92c3e67dd4 user_agent=kube-probe/1.23
guidoiaquinti commented 2 years ago

Thank you for the additional info. Is this error coming from a brand new installation or did you do an upgrade from a previous version? Can you please also check the migration job logs?

yog27ray commented 2 years ago

This is coming from a brand new installation. I am attaching the Async Migrations page screenshot.

Screenshot 2022-07-15 at 9 07 01 AM
guidoiaquinti commented 2 years ago

Sorry I meant the migration pod logs

On Fri, 15 Jul 2022 at 05:42, yog27ray @.***> wrote:

This is coming from a brand new installation. I am attaching the Async Migrations page screenshot. [image: Screenshot 2022-07-15 at 9 07 01 AM] https://user-images.githubusercontent.com/9247177/179145453-62d42838-98b6-44d4-899f-5e177a575742.png

— Reply to this email directly, view it on GitHub https://github.com/PostHog/charts-clickhouse/issues/485#issuecomment-1185142485, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6Z3GP5XF23XSOG6PW2C43VUDMZPANCNFSM53N7OF5A . You are receiving this because you commented.Message ID: @.***>

posthog-contributions-bot[bot] commented 2 years ago

This issue has 2238 words at 5 comments. Issues this long are hard to read or contribute to, and tend to take very long to reach a conclusion. Instead, why not:

  1. Write some code and submit a pull request! Code wins arguments
  2. Have a sync meeting to reach a conclusion
  3. Create a Request for Comments and submit a PR with it to the meta repo or product internal repo

Is this issue intended to be sprawling? Consider adding label epic or sprint to indicate this.

yog27ray commented 2 years ago

Hi below are the logs from migration pod

posthog-pgbouncer.posthog.svc.cluster.local (10.244.8.180:6543) open
posthog-posthog-postgresql.posthog.svc.cluster.local (10.244.12.127:5432) open
posthog-posthog-redis-master.posthog.svc.cluster.local (10.244.8.242:6379) open
posthog-posthog-kafka.posthog.svc.cluster.local (10.244.15.213:9092) open

🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻
Skipping async migrations setup. This is unsafe in production!
🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺

AXES: BEGIN LOG
AXES: BEGIN LOG
AXES: Using django-axes version 5.9.0
AXES: Using django-axes version 5.9.0
AXES: blocking by IP only.
AXES: blocking by IP only.
Report for 7d8d44d4929100cfcf2a4f572167a31e:
{'chart_version': '25.1.1',
 'cloud': 'gcp',
 'deployment': 'helm_gcp_ha',
 'deployment_type': 'helm',
 'hostname': 'posthog.xyz.abc',
 'ingress_type': 'nginx',
 'kube_version': 'v1.23.5-gke.1503',
 'operation': 'upgrade',
 'posthog_version': '1.37.1',
 'release_name': 'posthog',
 'release_revision': 2}

🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻
Skipping async migrations setup. This is unsafe in production!
🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺

AXES: BEGIN LOG
AXES: BEGIN LOG
AXES: Using django-axes version 5.9.0
AXES: Using django-axes version 5.9.0
AXES: blocking by IP only.
AXES: blocking by IP only.
Operations to perform:
  Apply all migrations: admin, auth, axes, contenttypes, ee, posthog, rest_hooks, sessions, social_django
Running migrations:
  No migrations to apply.

🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻
Skipping async migrations setup. This is unsafe in production!
🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺

AXES: BEGIN LOG
AXES: BEGIN LOG
AXES: Using django-axes version 5.9.0
AXES: Using django-axes version 5.9.0
AXES: blocking by IP only.
AXES: blocking by IP only.
✅ Migration successful

🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻
Skipping async migrations setup. This is unsafe in production!
🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺

AXES: BEGIN LOG
AXES: BEGIN LOG
AXES: Using django-axes version 5.9.0
AXES: Using django-axes version 5.9.0
AXES: blocking by IP only.
AXES: blocking by IP only.

🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻🔻
Skipping async migrations setup. This is unsafe in production!
🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺🔺

AXES: BEGIN LOG
AXES: BEGIN LOG
AXES: Using django-axes version 5.9.0
AXES: Using django-axes version 5.9.0
AXES: blocking by IP only.
AXES: blocking by IP only.
<BrokerConnection node_id=bootstrap-0 host=posthog-posthog-kafka:9092 <connecting> [IPv4 ('10.244.15.213', 9092)]>: connecting to posthog-posthog-kafka:9092 [('10.244.15.213', 9092) IPv4]
Probing node bootstrap-0 broker version
<BrokerConnection node_id=bootstrap-0 host=posthog-posthog-kafka:9092 <connecting> [IPv4 ('10.244.15.213', 9092)]>: Connection complete.
Broker version identified as 2.5.0
Set configuration api_version=(2, 5, 0) to skip auto check_version requests on startup
2022-07-12T05:34:06.936822Z [info     ] ✅ All ClickHouse nodes schema in sync [ee.management.commands.sync_replicated_schema] 
Closing the Kafka producer with 0 secs timeout.
Proceeding to force close the producer since pending requests could not be completed within timeout 0.
<BrokerConnection node_id=0 host=posthog-posthog-kafka-0.posthog-posthog-kafka-headless.posthog.svc.cluster.local:9092 <connecting> [IPv4 ('10.240.1.50', 9092)]>: connecting to posthog-posthog-kafka-0.posthog-posthog-kafka-headless.posthog.svc.cluster.local:9092 [('10.240.1.50', 9092) IPv4]
yog27ray commented 2 years ago

@guidoiaquinti is anything else needed from my side?

guidoiaquinti commented 2 years ago

From a first look it seems your ClickHouse installation is missing a table Table default.person_distinct_id2 doesn't exist. I'm not sure how you ended up like this but that's why the queries are failing.

I see you are running a sharded and replicated setup for ClickHouse. Did you added sharding and replicas after the initial un-sharded/un-replicated installation or did you start from scratch with this topology? cc @hazzadous

yog27ray commented 2 years ago

@guidoiaquinti I tried multiple times installations from scratch but every time it ends up in the same situation.

guidoiaquinti commented 2 years ago

We'll try to repro and debug on our end. Thank you for your report! 🙇

hazzadous commented 2 years ago

I'll add this to my list of Support tickets to handle and will get around to looking hopefully today.

hazzadous commented 2 years ago

I didn't get around to this last week. I'm unlikely to get around today either. @guidoiaquinti if you have a chance to repro... 🙏

Ok things that I think are important:

  1. if you're got PV driver setup to retain i.e. your GCP volumes from previous deployments will still be hanging around. So if there was some issue with the first deployment then it may be that subsequent deployments will have actually not been fresh deployments. There are lots of things around e.g. ClickHouse and ZooKeeper that can be odd in these cases.
  2. I haven't tested sharding much so it's very possible there is some issue here. I would 1. make sure PVs are cleaned up 2. deploy without sharding. You can then update the shard/replica counts and the Schema should be copied (although I've never tested this, only seen in the logs that it has tried to do this in the tests in CI, so take that with a grain of salt).
yog27ray commented 2 years ago

@hazzadous

  1. I had make sure that no previous deployment volumes were left.
  2. I deployed again without sharing and replica initially and that was working fine. Then updated the deployment with sharding and replica after that also everything is working fine.

There is a problem only when I do a fresh deployment with sharding and replica.

hazzadous commented 2 years ago

Got to know that the sharing upgrade worked. We’ll need to put some tests for the initial sharding

yog27ray commented 2 years ago

@hazzadous after some time of upgrade now I am again getting the same error.

DB::Exception: Table default.person_distinct_id2 doesn't exist. Stack trace:

I will try again with only the replica configured without a shard.

yog27ray commented 2 years ago

@hazzadous I tried with only replica configured and that is working perfectly fine. When I update the deployment with the shared config I again start getting the same error, and when I remove the shared the same deployment starts working fine again.

hazzadous commented 2 years ago

Ok I think this feature isn’t fully baked yet 😞

is it an option to go unsharded for now until we have sorted this?

yog27ray commented 2 years ago

@hazzadous for now I am not using the shared only replica. Will wait for the fix.

connorwang commented 2 years ago

I'm having exact same issue. Maybe related?

https://github.com/ClickHouse/ClickHouse/issues/19278#issuecomment-762834689