search

tryzealot / zealot

开源自部署移动应用、 macOS、Linux 和 Windows 应用分发平台，提供 iOS、Android SDK、fastlane 等丰富组件库 | Self-hosted Beta App Distribution for Android, iOS, macOS, Linux and Windows apps
https://zealot.ews.im
MIT License
1.08k stars 134 forks source link

NoMethodError: undefined method `split' for nil:NilClass #768

Closed Cleam closed 2 years ago

Cleam commented 2 years ago

部署方式 | How to Deploy

Docker (Default)

部署版本 | Version

latest

反代服务 | Reverse Proxy

nginx

已阅读文档 | Read the document

描述 | Description

一开始，系统只接入了一个IOS应用，运行比较稳定，后面新加入了一个ios应用，在发布较频繁的时候，很容易就把服务器搞崩溃，看了下后台任务监控信息如下（不知是不是失败重试导致服务器崩溃，目前我这里只能靠重启服务器解决）：

失败面板：

image

重试面板：

image

信息面板：

image

大佬，能不能帮忙看看，这一般是啥情况引起的？

问题日志 | Relevant log output

近100条日志（需要更多我再查）：

/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'
2022-05-11T08:43:59.530Z pid=319 tid=z3j class=TeardownJob jid=f5199d5e903817816eb46cff elapsed=7.482 INFO: fail
2022-05-11T08:43:59.530Z pid=319 tid=z3j WARN: {"context":"Job raised exception","job":{"retry":true,"queue":"app_parse","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"TeardownJob","args":[{"job_class":"TeardownJob","job_id":"9f66c66d-eea6-4f25-9cad-13ddfb939ecd","provider_job_id":null,"queue_name":"app_parse","priority":null,"arguments":[539,2],"executions":0,"exception_executions":{},"locale":"zh-CN","timezone":"Beijing","enqueued_at":"2022-05-11T06:11:10Z"}],"jid":"f5199d5e903817816eb46cff","created_at":1652249470.6100562,"enqueued_at":1652258632.04842,"error_message":"undefined method `split' for nil:NilClass","error_class":"NoMethodError","failed_at":1652258639.529939,"retry_count":8,"retried_at":1652254466.520403,"processor":"e1202b56b13a:319"},"jobstr":"{\"retry\":true,\"queue\":\"app_parse\",\"class\":\"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper\",\"wrapped\":\"TeardownJob\",\"args\":[{\"job_class\":\"TeardownJob\",\"job_id\":\"9f66c66d-eea6-4f25-9cad-13ddfb939ecd\",\"provider_job_id\":null,\"queue_name\":\"app_parse\",\"priority\":null,\"arguments\":[539,2],\"executions\":0,\"exception_executions\":{},\"locale\":\"zh-CN\",\"timezone\":\"Beijing\",\"enqueued_at\":\"2022-05-11T06:11:10Z\"}],\"jid\":\"f5199d5e903817816eb46cff\",\"created_at\":1652249470.6100562,\"enqueued_at\":1652258632.04842,\"error_message\":\"undefined method `split' for nil:NilClass\",\"error_class\":\"NoMethodError\",\"failed_at\":1652249476.7107005,\"retry_count\":8,\"retried_at\":1652254466.520403}"}
2022-05-11T08:43:59.530Z pid=319 tid=z3j WARN: NoMethodError: undefined method `split' for nil:NilClass
2022-05-11T08:43:59.531Z pid=319 tid=z3j WARN: /app/app/services/teardown_service.rb:81:in `block in process_ios'
/app/app/services/teardown_service.rb:80:in `each'
/app/app/services/teardown_service.rb:80:in `each_with_object'
/app/app/services/teardown_service.rb:80:in `process_ios'
/app/app/services/teardown_service.rb:34:in `process'
/app/app/services/teardown_service.rb:20:in `call'
/app/app/services/application_service.rb:5:in `call'
/app/app/jobs/teardown_job.rb:9:in `perform'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/execution.rb:48:in `block in perform_now'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/sentry-rails-4.8.1/lib/sentry/rails/active_job.rb:16:in `block (2 levels) in included'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `instance_exec'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/i18n-1.8.11/lib/i18n.rb:314:in `with_locale'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `instance_exec'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/core_ext/time/zones.rb:66:in `use_zone'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/timezones.rb:9:in `block (2 levels) in <module:Timezones>'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `instance_exec'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/instrumentation.rb:21:in `block in instrument'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/notifications.rb:203:in `block in instrument'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/notifications.rb:203:in `instrument'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/instrumentation.rb:31:in `instrument'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/instrumentation.rb:14:in `block (2 levels) in <module:Instrumentation>'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `instance_exec'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/logging.rb:24:in `tag_logger'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/logging.rb:15:in `block (2 levels) in <module:Logging>'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `instance_exec'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:137:in `run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/execution.rb:47:in `perform_now'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/execution.rb:25:in `block in execute'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/railtie.rb:47:in `block (4 levels) in <class:Railtie>'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/reloader.rb:72:in `block in wrap'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/execution_wrapper.rb:84:in `wrap'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/reloader.rb:71:in `wrap'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/railtie.rb:46:in `block (3 levels) in <class:Railtie>'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `instance_exec'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/callbacks.rb:137:in `run_callbacks'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/execution.rb:23:in `execute'
/app/vendor/bundle/ruby/2.7.0/gems/activejob-6.1.4.4/lib/active_job/queue_adapters/sidekiq_adapter.rb:42:in `perform'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:196:in `execute_job'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:138:in `block in invoke'
/app/vendor/bundle/ruby/2.7.0/gems/sentry-sidekiq-4.8.1/lib/sentry/sidekiq/sentry_context_middleware.rb:7:in `call'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-failures-1.0.1/lib/sidekiq/failures/middleware.rb:9:in `call'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:143:in `invoke'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:163:in `block in process'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:112:in `local'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/rails.rb:14:in `block in call'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/reloader.rb:72:in `block in wrap'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/app/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/reloader.rb:71:in `wrap'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/rails.rb:13:in `call'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'
/app/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'
icyleaf commented 2 years ago

麻烦可以问下 iOS 应用中 CFBundleURLSchemes 是如何设置的值？

icyleaf commented 2 years ago

我先处理这个问题，等一会 https://github.com/tryzealot/zealot/actions/runs/2306089077 跑完更新下 nightly tag 后就应该好了

Cleam commented 2 years ago

响应这么快，感谢大佬，我晚点试试nightly。

Cleam commented 2 years ago

麻烦可以问下 iOS 应用中 CFBundleURLSchemes 是如何设置的值？

新增的IOS应用，CFBundleURLSchemes 为空。

Cleam commented 2 years ago

@icyleaf 请教下大佬，我们已经将空的CFBundleURLSchemes值处理掉了，没有报错了，但是发包频率高的情况下，还是会把服务器卡死（目前只接了2个IOS应用），不知道这个要怎么排查问题？或者大佬是否知道有可能是哪些原因导致？

icyleaf commented 2 years ago

先确认下打包机器的硬件规格，CPU 和 内存 只要把这俩解决了都不是问题

Cleam commented 2 years ago

先确认下打包机器的硬件规格，CPU 和 内存 只要把这俩解决了都不是问题

目前的服务器:

icyleaf commented 2 years ago

那不应该，我 4 核 2G 内存都能稳定跑几年，iOS 不会特别吃内存，Android 解析会相对比较吃内存的，机器还有其他什么吃资源的服务吗或者部署 zealot 有限制资源吗

Cleam commented 2 years ago

那不应该，我 4 核 2G 内存都能稳定跑几年，iOS 不会特别吃内存，Android 解析会相对比较吃内存的，机器还有其他什么吃资源的服务吗或者部署 zealot 有限制资源吗

不知大佬是否有对docker配置进行优化？我这里docker部署了几个服务，也部署了几年，但是自从部署了zealot服务之后，不定时的就会将内存占满，导致服务崩溃，如下内存占满图：

image

我尝试过一个一个的停掉docker服务，但是当我停掉zealot这个服务的时候，内存一下子就降下来了。尴尬的是我也不知道这个问题出在哪。

icyleaf commented 2 years ago

docker info 看看

Cleam commented 2 years ago

@icyleaf docker info 如下：

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.9.0)

Server:
 Containers: 14
  Running: 8
  Paused: 0
  Stopped: 6
 Images: 20
 Server Version: 20.10.11
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.15.6-1.el7.elrepo.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.63GiB
 Name: 10-42-1-77
 ID: E3QL:COS7:2IJW:2QXU:GYPV:B7UO:6T2N:AYDJ:X5YU:7SME:LV6S:YCWU
 Docker Root Dir: /data/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://*****.mirror.aliyuncs.com/
  https://hub-mirror.c.163.com/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 192.169.0.0/16, Size: 24

我昨晚看了下，内存占用保持在了50%左右（感觉内存占用每天会不断增加而没有释放），只能通过docker-compose stop再start才会释放，restart都没有释放。

image
icyleaf commented 2 years ago

把 storage driver 改成 overlay2 试试吧 https://docs.docker.com/storage/storagedriver/overlayfs-driver/

icyleaf commented 2 years ago

我们的服务部署在阿里云的 k8s 上面使用 0.25 core 和 512M 内存跑的那么久也没啥问题

iShot_2022-05-20_11 43 17
Cleam commented 2 years ago

@icyleaf 系统暂不支持 overlay2 暂时只能写个定时器 ，定时重启服务。

icyleaf commented 2 years ago

建议不要使用 CentOS 换个其他主流系统吧 😂 我先关闭了

Cleam commented 2 years ago

ok

Cleam commented 2 years ago

@icyleaf 昨天将docker容器内存和cpu做了限制之后，发现没有出现内存暴涨的情况，我将定时重启服务的任务也暂时去掉了，打算再观察一阵子，不过看内存占用情况，明显好转了：

image

内存限制（之前是没做限制的）：

image

image

icyleaf commented 2 years ago

感谢经过观察的反馈，后续我会把你情况整理并汇总到 Q&A 里面