Server Crash #10704

Closed Aviortheking closed 4 years ago

Gitea version (or commit ref): Lastest Docker build
Git version: Docker VM
Operating system: Ubuntu 18.04.4 LTS
Database (use [x]):
- [ ] PostgreSQL
- [x] MySQL
- [ ] MSSQL
- [ ] SQLite
Can you reproduce the bug at https://try.gitea.io:
- [ ] Yes (provide example URL)
- [x] No
- [ ] Not relevant
Log gist: gitea.log (Sorry for length)
- interesting lines at 34290 where it received a 15 error from sshd
- after that Server Started

Description

Gitea has a memory leak and make my server crash in 1-2 minutes if I don't stop it

It started yesterday in the evening and since then I just can't use it anymore

for informations :

My biggest repo in size is Gitea itself
My biggest repo in number of files is https://github.com/tcgdex/distribution

I tried :

Updating the docker image
Redownloading the docker image

...

Screenshots

Memory Screenshot of the spike (the two other are another VM) Screenshot from 2020-03-10 20-18-34

Are there any errors in log?

depending on what errors you want I got this : (grepping [E] text)

server_1  | 2020/03/10 18:41:27 routers/init.go:73:initDBEngine() [E] ORM engine initialization attempt #1/10 failed. Error: dial tcp 172.18.0.2:3306: connect: connection refused
server_1  | 2020/03/10 18:41:30 routers/init.go:73:initDBEngine() [E] ORM engine initialization attempt #2/10 failed. Error: dial tcp 172.18.0.2:3306: connect: connection refused
server_1  | 2020/03/10 18:41:33 routers/init.go:73:initDBEngine() [E] ORM engine initialization attempt #3/10 failed. Error: dial tcp 172.18.0.2:3306: connect: connection refused
server_1  | 2020/03/10 18:42:44 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:42:44 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:42:51 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:42:51 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:43:26 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:43:26 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:43:56 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:43:57 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:46:48 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:46:48 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:46:57 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:46:57 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:47:02 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:47:02 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:47:18 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:47:18 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:47:26 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:47:26 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:47:29 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:47:29 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:47:37 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:47:37 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: vendor, keyid: 0]
server_1  | 2020/03/10 18:47:59 routers/repo/http.go:103:HTTP() [E] GetUserByName: user does not exist [uid: 0, name: , keyid: 0]

another here

[32mserver_1  |[0m Error loading shared library libcrypto.so.1.1: Out of memory (needed by /usr/sbin/sshd)
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_div: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_DigestInit_ex: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_set0_factors: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_generate_parameters_ex: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_PKEY_set1_DSA: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_KEY_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_SIG_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_set_flags: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_CipherInit: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ERR_get_error: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_CIPHER_CTX_iv_length: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_MD_CTX_md: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_CTX_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_PKEY_get1_EC_KEY: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_CIPHER_CTX_key_length: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ECDSA_SIG_get0: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DH_get0_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DH_size: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_POINT_is_at_infinity: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_SIG_set0: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_GROUP_new_by_curve_name: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RAND_seed: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RAND_poll: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DH_compute_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_PKEY_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: AES_encrypt: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_do_sign: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_KEY_get0_public_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_set_word: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_bin2bn: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_GROUP_method_of: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_CIPHER_CTX_ctrl: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_GROUP_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_POINT_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_PKEY_base_id: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_KEY_set_group: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_KEY_set_public_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_set0_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_PKEY_get1_DSA: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_generate_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_num_bits: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_is_negative: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_generate_key_ex: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_DigestUpdate: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DH_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_sub: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_KEY_get0_group: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: OpenSSL_version_num: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_get0_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_cmp: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_set0_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_get0_factors: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_MD_block_size: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_size: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_DigestFinal_ex: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_POINT_point2oct: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_PKEY_set1_RSA: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_PKEY_set1_EC_KEY: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_dup: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DH_set0_pqg: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_KEY_generate_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_clear_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: MD5_Final: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: MD5_Update: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BIO_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_POINT_get_affine_coordinates_GFp: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_POINT_clear_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_is_bit_set: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_hex2bn: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ECDSA_do_verify: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: AES_set_encrypt_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_set0_pqg: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ECDSA_SIG_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: PEM_read_bio_PrivateKey: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_MD_CTX_copy_ex: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_POINT_cmp: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_print_fp: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_KEY_new_by_curve_name: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DH_generate_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_MD_CTX_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BIO_s_mem: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_GROUP_get_curve_name: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_public_decrypt: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_value_one: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ENGINE_load_builtin_engines: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_Digest: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_CIPHER_CTX_iv: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_CIPHER_CTX_set_key_length: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_CIPHER_CTX_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_CTX_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_KEY_set_private_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_GROUP_get_order: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: PEM_write_bio_ECPrivateKey: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_POINT_mul: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ERR_peek_last_error: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ECDH_compute_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ERR_peek_error: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_GROUP_cmp: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BIO_write: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_aes_128_cbc: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: PEM_write_bio_RSAPrivateKey: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: PEM_write_bio_DSAPrivateKey: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_sign: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: MD5_Init: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DH_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_SIG_get0: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_GROUP_get_degree: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_GROUP_set_asn1_flag: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_set0_crt_params: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_do_verify: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_get0_pqg: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_POINT_oct2point: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_CIPHER_CTX_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_MD_CTX_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DH_set_length: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ENGINE_register_all_complete: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_POINT_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RAND_bytes: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_Cipher: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_PKEY_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_blinding_on: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_CTX_get: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_get0_crt_params: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_bn2bin: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ECDSA_SIG_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BIO_ctrl: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ECDSA_SIG_set0: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BIO_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RAND_status: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: OPENSSL_init_crypto: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_PKEY_get1_RSA: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: ECDSA_do_sign: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_KEY_set_asn1_flag: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_CIPHER_CTX_iv_noconst: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: OpenSSL_version: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_SIG_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_METHOD_get_field_type: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EC_KEY_get0_private_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DSA_new: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_CTX_start: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: RSA_get0_key: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: DH_get0_pqg: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: BN_free: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: PEM_write_bio_PrivateKey: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_des_ede3_cbc: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_aes_192_cbc: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_aes_256_cbc: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_aes_256_cbc: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_aes_128_ctr: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_aes_192_ctr: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_aes_256_ctr: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_aes_128_gcm: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_aes_256_gcm: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_md5: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_sha1: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_sha256: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_sha384: symbol not found
[32mserver_1  |[0m Error relocating /usr/sbin/sshd: EVP_sha512: symbol not found
[33mdb_1      |[0m 2020-03-10T18:50:24.893873Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 14010ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
[32mserver_1  |[0m Received signal 15; terminating.

Could you confirm your docker image is before or after #10697 ?

I'm after the commit

It's a out of memory error. loading shared library libcrypto.so.1.1: Out of memory (needed by /usr/sbin/sshd) when start sshd on gitea docker. But cannot find why that occuried from logs.

I don't know by what sorcery it happened but now it don't happen anymore :smile: !

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

go-gitea / gitea

Server Crash #10704

Description

Screenshots