search

SigNoz / signoz

SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
https://signoz.io
Other
18.92k stars 1.23k forks source link

unknown time zone Asia/Shanghai #1150

Closed myoss closed 2 years ago

myoss commented 2 years ago

Bug description

docker run  -v /root/otel/config.yaml:/etc/otel/config.yaml --rm -it signoz-otelcontribcol:0.43.0-0.1
Error: cannot build exporters: error creating clickhouse exporter: error connecting to primary db: could not load time location: unknown time zone Asia/Shanghai
2022/05/17 09:45:28 collector server run finished with error: cannot build exporters: error creating clickhouse exporter: error connecting to primary db: could notload time location: unknown time zone Asia/Shanghai
welcome[bot] commented 2 years ago

Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.

myoss commented 2 years ago

use external /usr/local/Cellar/go/1.17/libexec/lib/time/zoneinfo.zip

    spec:
      containers:
        - name: demo
          image: busybox:lastest
          volumeMounts:
            - name: host-time
              mountPath: /opt/zoneinfo.zip
          env:
            - name: ZONEINFO
              value: /opt/zoneinfo.zip
      volumes:
        - name: host-time
          hostPath:
            path: /opt/zoneinfo.zip
prashant-shahi commented 2 years ago

Hey @myoss!

I am able to see Asia/Shanghai in the clickhouse image that is used in SigNoz. Are you using external Clickhouse? if yes, please share information like docker image and version.

I am tried to replicate on my machine but I wasn't able to.

Can you share more details on how it can be replicated? Also, do share what you have done differently like any OtelCollector or Clickhouse config change or any deployment related changes?

myoss commented 2 years ago

used external clickhouse, deployed by other teams, our server time zone is Asia/Shanghai

[super@demo ~]$ docker version
Client:
 Version:         1.13.1
 API version:     1.26

[super@demo ~]$ date -R
Thu, 19 May 2022 18:24:17 +0800
prashant-shahi commented 2 years ago

@myoss my bad for not being clear..

I meant to ask for the ClickHouse version and if you have the idea about the Clickhouse Image. If you are using public Clickhouse image, can you please share that.

Also, if possible, can you share the OtelCollector configuration that you used?

myoss commented 2 years ago 
[root@demo ~]# ls /tmp/clickhouse-common-static-22.3.3.44/usr/bin/
clickhouse                      config.xml                      format_schemas/                 store/                          user_files/
clickhouse-extract-from-config  data/                           metadata/                       tmp/                            user_scripts/
clickhouse-library-bridge       dictionaries_lib/               metadata_dropped/               t.sh                            uuid
clickhouse-odbc-bridge          flags/                          preprocessed_configs/           user_defined/
[root@demo ~]# cd /tmp/clickhouse-common-static-22.3.3.44/usr/bin/
[root@demo bin]# ./clickhouse server -C config.xml
Processing configuration file 'config.xml'.
Logging trace to console
2022.05.20 09:39:56.849952 [ 10590 ] {} <Information> SentryWriter: Sending crash reports is disabled
2022.05.20 09:39:56.858155 [ 10590 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB
2022.05.20 09:39:57.380524 [ 10590 ] {} <Information> : Starting ClickHouse 22.3.3.44 with revision 54460, build id: F9D3C2B8666BEF5D, PID 10590
2022.05.20 09:39:57.380596 [ 10590 ] {} <Information> Application: starting up
2022.05.20 09:39:57.380614 [ 10590 ] {} <Information> Application: OS name: Linux, version: 3.10.0-957.27.2.el7.x86_64, architecture: x86_64
2022.05.20 09:39:57.857484 [ 10590 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/thermal
2022.05.20 09:39:57.857526 [ 10590 ] {} <Trace> AsynchronousMetrics: Scanning /sys/block
2022.05.20 09:39:57.857610 [ 10590 ] {} <Trace> AsynchronousMetrics: Scanning /sys/devices/system/edac
2022.05.20 09:39:57.857633 [ 10590 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/hwmon
2022.05.20 09:39:58.164354 [ 10590 ] {} <Warning> Application: Calculated checksum of the binary: FD20C2FC24F8B8996C15BF97FA841B03. There is no information about the reference checksum.
2022.05.20 09:39:58.164505 [ 10590 ] {} <Trace> Application: Will do mlock to prevent executable memory from being paged out. It may take a few seconds.
2022.05.20 09:39:58.185478 [ 10590 ] {} <Trace> Application: The memory map of clickhouse executable has been mlock'ed, total 275.48 MiB
2022.05.20 09:39:58.186145 [ 10590 ] {} <Debug> Application: rlimit on number of file descriptors is 65535
2022.05.20 09:39:58.186162 [ 10590 ] {} <Debug> Application: Initializing DateLUT.
2022.05.20 09:39:58.186168 [ 10590 ] {} <Trace> Application: Initialized DateLUT with time zone 'Asia/Shanghai'.
2022.05.20 09:39:58.186374 [ 10590 ] {} <Debug> Application: Setting up ./tmp/ to store temporary data in it
2022.05.20 09:39:58.186438 [ 10590 ] {} <Debug> Application: Initiailizing interserver credentials.
2022.05.20 09:39:58.189249 [ 10590 ] {} <Debug> ConfigReloader: Loading config 'config.xml'
Processing configuration file 'config.xml'.
Saved preprocessed configuration to './preprocessed_configs/config.xml'.
2022.05.20 09:39:58.189491 [ 10590 ] {} <Debug> ConfigReloader: Loaded config 'config.xml', performing update on configuration
2022.05.20 09:39:58.189686 [ 10590 ] {} <Information> Application: Setting max_server_memory_usage was set to 6.64 GiB (7.37 GiB available * 0.90 max_server_memory_usage_to_ram_ratio)
2022.05.20 09:39:58.190003 [ 10590 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again.
2022.05.20 09:39:58.190025 [ 10590 ] {} <Debug> ConfigReloader: Loaded config 'config.xml', performed update on configuration
2022.05.20 09:39:58.192947 [ 10590 ] {} <Debug> ConfigReloader: Loading config 'config.xml'
Processing configuration file 'config.xml'.
Saved preprocessed configuration to './preprocessed_configs/config.xml'.
2022.05.20 09:39:58.193166 [ 10590 ] {} <Debug> ConfigReloader: Loaded config 'config.xml', performing update on configuration
2022.05.20 09:39:58.193779 [ 10590 ] {} <Debug> ConfigReloader: Loaded config 'config.xml', performed update on configuration
2022.05.20 09:39:58.194046 [ 10590 ] {} <Debug> Access(user directories): Added users.xml access storage 'users.xml', path: config.xml
2022.05.20 09:39:58.194073 [ 10590 ] {} <Information> Application: Uncompressed cache size was lowered to 3.69 GiB because the system has low amount of memory
2022.05.20 09:39:58.194776 [ 10590 ] {} <Information> Context: Initialized background executor for merges and mutations with num_threads=16, num_tasks=32
2022.05.20 09:39:58.195308 [ 10590 ] {} <Information> Context: Initialized background executor for move operations with num_threads=8, num_tasks=8
2022.05.20 09:39:58.195716 [ 10590 ] {} <Information> Context: Initialized background executor for fetches with num_threads=8, num_tasks=8
2022.05.20 09:39:58.196194 [ 10590 ] {} <Information> Context: Initialized background executor for common operations (e.g. clearing old parts) with num_threads=8, num_tasks=8
2022.05.20 09:39:58.196376 [ 10590 ] {} <Information> Application: Mark cache size was lowered to 3.69 GiB because the system has low amount of memory
2022.05.20 09:39:58.196418 [ 10590 ] {} <Information> Application: Loading user defined objects from ./
2022.05.20 09:39:58.196430 [ 10590 ] {} <Debug> UserDefinedSQLObjectsLoader: loading user defined objects
2022.05.20 09:39:58.196470 [ 10590 ] {} <Debug> Application: Loaded user defined objects
2022.05.20 09:39:58.196477 [ 10590 ] {} <Information> Application: Loading metadata from ./
2022.05.20 09:39:58.199601 [ 10590 ] {} <Information> DatabaseAtomic (system): Metadata processed, database system has 0 tables and 0 dictionaries in total.
2022.05.20 09:39:58.199617 [ 10590 ] {} <Information> TablesLoader: Parsed metadata of 0 tables in 1 databases in 0.000421885 sec
2022.05.20 09:39:58.199624 [ 10590 ] {} <Information> TablesLoader: Loading 0 tables with 0 dependency level
2022.05.20 09:39:58.199635 [ 10590 ] {} <Debug> SystemLog: Not creating system.query_log since corresponding section 'query_log' is missing from config
2022.05.20 09:39:58.199645 [ 10590 ] {} <Debug> SystemLog: Not creating system.query_thread_log since corresponding section 'query_thread_log' is missing from config
2022.05.20 09:39:58.199654 [ 10590 ] {} <Debug> SystemLog: Not creating system.part_log since corresponding section 'part_log' is missing from config
2022.05.20 09:39:58.199662 [ 10590 ] {} <Debug> SystemLog: Not creating system.trace_log since corresponding section 'trace_log' is missing from config
2022.05.20 09:39:58.199668 [ 10590 ] {} <Debug> SystemLog: Not creating system.crash_log since corresponding section 'crash_log' is missing from config
2022.05.20 09:39:58.199676 [ 10590 ] {} <Debug> SystemLog: Not creating system.text_log since corresponding section 'text_log' is missing from config
2022.05.20 09:39:58.199683 [ 10590 ] {} <Debug> SystemLog: Not creating system.metric_log since corresponding section 'metric_log' is missing from config
2022.05.20 09:39:58.199692 [ 10590 ] {} <Debug> SystemLog: Not creating system.asynchronous_metric_log since corresponding section 'asynchronous_metric_log' is missing from config
2022.05.20 09:39:58.199699 [ 10590 ] {} <Debug> SystemLog: Not creating system.opentelemetry_span_log since corresponding section 'opentelemetry_span_log' is missing from config
2022.05.20 09:39:58.199709 [ 10590 ] {} <Debug> SystemLog: Not creating system.query_views_log since corresponding section 'query_views_log' is missing from config
2022.05.20 09:39:58.199716 [ 10590 ] {} <Debug> SystemLog: Not creating system.zookeeper_log since corresponding section 'zookeeper_log' is missing from config
2022.05.20 09:39:58.199723 [ 10590 ] {} <Debug> SystemLog: Not creating system.session_log since corresponding section 'session_log' is missing from config
2022.05.20 09:39:58.208653 [ 10590 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal.
2022.05.20 09:39:58.211292 [ 10590 ] {} <Information> DatabaseAtomic (default): Metadata processed, database default has 3 tables and 0 dictionaries in total.
2022.05.20 09:39:58.213225 [ 10590 ] {} <Information> DatabaseAtomic (signoz_traces): Metadata processed, database signoz_traces has 7 tables and 0 dictionaries intotal.
2022.05.20 09:39:58.213240 [ 10590 ] {} <Information> TablesLoader: Parsed metadata of 10 tables in 2 databases in 0.003470319 sec
2022.05.20 09:39:58.213248 [ 10590 ] {} <Information> TablesLoader: Loading 10 tables with 0 dependency level
2022.05.20 09:39:58.214467 [ 10643 ] {} <Debug> default.signoz_error_index (a6c0fd3b-b164-4a25-99e6-c481141fb0e5): Loading data parts
2022.05.20 09:39:58.214701 [ 10643 ] {} <Debug> default.signoz_error_index (a6c0fd3b-b164-4a25-99e6-c481141fb0e5): There are no data parts
2022.05.20 09:39:58.215102 [ 10642 ] {} <Debug> default.signoz_index (a4899a88-2ba0-4a24-a5a3-bfa707b037f6): Loading data parts
2022.05.20 09:39:58.215172 [ 10642 ] {} <Debug> default.signoz_index (a4899a88-2ba0-4a24-a5a3-bfa707b037f6): There are no data parts
2022.05.20 09:39:58.215953 [ 10642 ] {} <Debug> signoz_traces.samples (bad041dd-2264-4ee9-8d28-76954133f80a): Loading data parts
2022.05.20 09:39:58.216099 [ 10643 ] {} <Debug> signoz_traces.durationSort (1cbadc36-0673-4eb7-ac69-4c3ad16a5c38): Loading data parts
2022.05.20 09:39:58.216364 [ 10643 ] {} <Debug> signoz_traces.durationSort (1cbadc36-0673-4eb7-ac69-4c3ad16a5c38): There are no data parts
2022.05.20 09:39:58.217274 [ 10643 ] {} <Debug> signoz_traces.signoz_error_index (bfa067cd-108e-462d-b822-1099366cd127): Loading data parts
2022.05.20 09:39:58.217342 [ 10643 ] {} <Debug> signoz_traces.signoz_error_index (bfa067cd-108e-462d-b822-1099366cd127): There are no data parts
2022.05.20 09:39:58.218001 [ 10643 ] {} <Debug> signoz_traces.signoz_spans (5a6cad7a-904d-408f-8a78-0fbbd4049e31): Loading data parts
2022.05.20 09:39:58.218068 [ 10643 ] {} <Debug> signoz_traces.signoz_spans (5a6cad7a-904d-408f-8a78-0fbbd4049e31): There are no data parts
2022.05.20 09:39:58.218586 [ 10642 ] {} <Debug> signoz_traces.samples (bad041dd-2264-4ee9-8d28-76954133f80a): Loaded data parts (2 items)
2022.05.20 09:39:58.218677 [ 10643 ] {} <Debug> signoz_traces.time_series (33947d65-8f2d-4075-92b7-4c725d36cb7f): Loading data parts
2022.05.20 09:39:58.220497 [ 10642 ] {} <Debug> signoz_traces.signoz_index_v2 (6924023a-94ed-4597-919f-df9950bacfa4): Loading data parts
2022.05.20 09:39:58.220581 [ 10642 ] {} <Debug> signoz_traces.signoz_index_v2 (6924023a-94ed-4597-919f-df9950bacfa4): There are no data parts
2022.05.20 09:39:58.220961 [ 10643 ] {} <Debug> signoz_traces.time_series (33947d65-8f2d-4075-92b7-4c725d36cb7f): Loaded data parts (1 items)
2022.05.20 09:39:58.221095 [ 10590 ] {} <Information> DatabaseAtomic (default): Starting up tables.
2022.05.20 09:39:58.221157 [ 10642 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads
2022.05.20 09:39:58.227664 [ 10590 ] {} <Information> DatabaseAtomic (signoz_traces): Starting up tables.
2022.05.20 09:39:58.231833 [ 10590 ] {} <Information> DatabaseAtomic (system): Starting up tables.
2022.05.20 09:39:58.234427 [ 10590 ] {} <Debug> Application: Loaded metadata.
2022.05.20 09:39:58.353779 [ 10590 ] {} <Trace> MySQLHandlerFactory: Failed to create SSL context. SSL will be disabled. Error: Poco::Exception. Code: 1000, e.code() = 0, SSL Exception: Configuration error: no certificate file has been specified (version 22.3.3.44 (official build))
2022.05.20 09:39:58.353907 [ 10590 ] {} <Trace> MySQLHandlerFactory: Failed to read RSA key pair from server certificate. Error: Code: 139. DB::Exception: Certificate file is not set. (NO_ELEMENTS_IN_CONFIG) (version 22.3.3.44 (official build))
2022.05.20 09:39:58.353917 [ 10590 ] {} <Trace> MySQLHandlerFactory: Generating new RSA key pair.
2022.05.20 09:39:58.397179 [ 10590 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again.
2022.05.20 09:39:58.397212 [ 10590 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 11.87 MiB, peak 15.47 MiB, will set to 416.73 MiB (RSS), difference: 404.87 MiB
2022.05.20 09:39:58.397822 [ 10590 ] {} <Information> DNSCacheUpdater: Update period 15 seconds
2022.05.20 09:39:58.397888 [ 10590 ] {} <Information> Application: Available RAM: 7.37 GiB; physical cores: 2; logical cores: 4.
2022.05.20 09:39:58.397893 [ 10684 ] {} <Debug> DNSResolver: Updating DNS cache
2022.05.20 09:39:58.397968 [ 10684 ] {} <Debug> DNSResolver: Updated DNS cache
2022.05.20 09:39:58.398009 [ 10590 ] {} <Information> Application: Listening for http://172.20.200.120:8123
2022.05.20 09:39:58.398063 [ 10590 ] {} <Information> Application: Listening for native protocol (tcp): 172.20.200.120:9000
2022.05.20 09:39:58.398104 [ 10590 ] {} <Information> Application: Listening for MySQL compatibility protocol: 172.20.200.120:9004
2022.05.20 09:39:58.398110 [ 10590 ] {} <Information> Application: Ready for connections.
2022.05.20 09:40:13.398054 [ 10720 ] {} <Debug> DNSResolver: Updating DNS cache
2022.05.20 09:40:13.398084 [ 10720 ] {} <Debug> DNSResolver: Updated DNS cache

[root@demo bin]# cat config.xml
<clickhouse>
    <logger>
        <level>trace</level>
        <console>true</console>
    </logger>

    <listen_host>172.20.200.120</listen_host>
    <http_port>8123</http_port>
    <tcp_port>9000</tcp_port>
    <mysql_port>9004</mysql_port>

    <path>./</path>

    <uncompressed_cache_size>8589934592</uncompressed_cache_size>
    <mark_cache_size>5368709120</mark_cache_size>
    <mlock_executable>true</mlock_executable>

    <users>
        <default>
            <password/>

            <networks>
                <ip>::/0</ip>
            </networks>

            <profile>default</profile>
            <quota>default</quota>
            <access_management>1</access_management>
        </default>
    </users>

    <profiles>
        <default/>
    </profiles>

    <quotas>
        <default/>
    </quotas>
</clickhouse>

[root@demo otel]# cat /root/otel/config.yaml
exporters:
  clickhouse:
    datasource: tcp://172.20.200.120:9000/?database=signoz_traces
  clickhousemetricswrite:
    endpoint: tcp://172.20.200.120:9000/?database=signoz_traces
    resource_to_telemetry_conversion:
      enabled: true
  prometheus:
    endpoint: 0.0.0.0:8889
extensions:
  health_check: {}
  zpages: {}
processors:
  batch:
    send_batch_size: 1000
    timeout: 10s
  signozspanmetrics/prometheus:
    dimensions_cache_size: 10000
    latency_histogram_buckets:
    - 100us
    - 1ms
    - 2ms
    - 6ms
    - 10ms
    - 50ms
    - 100ms
    - 250ms
    - 500ms
    - 1000ms
    - 1400ms
    - 2000ms
    - 5s
    - 10s
    - 20s
    - 40s
    - 60s
    metrics_exporter: prometheus
receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: null
      disk: null
      filesystem: null
      load: null
      memory: null
      network: null
  jaeger:
    protocols:
      grpc: null
      thrift_http: null
  otlp:
    protocols:
      grpc: null
      http: null
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: localhost:12345
service:
  extensions:
  - health_check
  - zpages
  pipelines:
    metrics:
      exporters:
      - clickhousemetricswrite
      processors:
      - batch
      receivers:
      - otlp
      - hostmetrics
    metrics/spanmetrics:
      exporters:
      - prometheus
      receivers:
      - otlp/spanmetrics
    traces:
      exporters:
      - clickhouse
      processors:
      - signozspanmetrics/prometheus
      - batch
      receivers:
      - jaeger
      - otlp
YoKv commented 2 years ago

in my case : external clickhouse version 22.6.3.35 /etc/clickhouse-server/config.d/config.xml

<?xml version="1.0"?>
<clickhouse>
    <listen_host>0.0.0.0</listen_host>
    <timezone>Asia/Shanghai</timezone>
</clickhouse>

image 'signoz/query-service:0.10.0'

error log

2022-08-01T02:42:22.972Z        ERROR   clickhouseReader/reader.go:102  failed to initialize ClickHouse: error connecting to primary db: could not load time location: unknown time zone Asia/Shanghai
go.signoz.io/query-service/app/clickhouseReader.NewReader
        /go/src/github.com/signoz/signoz/pkg/query-service/app/clickhouseReader/reader.go:102
go.signoz.io/query-service/app.NewServer
        /go/src/github.com/signoz/signoz/pkg/query-service/app/server.go:85
main.main
        /go/src/github.com/signoz/signoz/pkg/query-service/main.go:66
runtime.main
        /usr/local/go/src/runtime/proc.go:255
myoss commented 2 years ago

@YoKv fix in local, solution:

# https://hub.docker.com/r/signoz/query-service/tags
FROM signoz/query-service:0.10.0

COPY zoneinfo.zip /zoneinfo.zip
ENV ZONEINFO=/zoneinfo.zip

ENTRYPOINT ["./query-service"]
CMD ["-config" "/root/config/prometheus.yml"]
docker build -t signoz/query-service:0.10.0 -f ./dockerfile-query-service .
docker push signoz/query-service:0.10.0
prashant-shahi commented 2 years ago

@myoss @YoKv Thank you for raising this issue and solutions in detail.

This issue has been resolved in develop for query-service. The fix will be included in next patch release soon. If you want to check it out, feel free to use develop docker image tag.

And for otel-collector, fixed is out with latest release 0.45.1-1.2