elastic search won't start

[loigu]

After upgrading the Docker image to onlyoffice/communityserver ID 7722d589c509 I noticed that the elastic search won't start. Search is not working.

[FAILED] Failed to start Elasticsearch. See 'systemctl status elasticsearch.service' for details.

root@onlyoffice:/# journalctl -xe -u elasticsearch -- Logs begin at Sat 2020-12-05 04:43:44 UTC, end at Sat 2020-12-05 05:00:17 UTC. -- Dec 05 04:43:44 onlyoffice systemd[1]: Starting Elasticsearch... -- Subject: Unit elasticsearch.service has begun start-up -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit elasticsearch.service has begun starting up. Dec 05 04:43:52 onlyoffice elasticsearch[389]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Dec 05 04:44:06 onlyoffice systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE Dec 05 04:44:06 onlyoffice systemd[1]: elasticsearch.service: Failed with result 'exit-code'. Dec 05 04:44:06 onlyoffice systemd[1]: Failed to start Elasticsearch. -- Subject: Unit elasticsearch.service has failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit elasticsearch.service has failed. -- -- The result is RESULT.

there is only gc.log in /var/log/elasticsearch - does not seem to tell much.

[Carazyda]

Hello @loigu how did you run onlyoffice docker version? Manually or used our script? And how did you update? If you run manually please add a new volume parameter to your docker run command -v /sys/fs/cgroup:/sys/fs/cgroup:ro. If that not helped - look at /var/log/onlyoffice/Index folder for the errors.

[loigu]

Dear @Carazyda I use my own script to control the docker container. I updated by "pull". I do have done all the updates needed with version 11.0 so I have the cgroup volume attached.

docker run -dit --restart=always --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro --tmpfs /tmp --tmpfs /run --tmpfs /run/lock --name onlyoffice --hostname onlyoffice --net macvlan --ip 192.168.0.36 --mac-address 0a:0a:0a:0a:0a:36 -e MYSQL_SERVER_HOST=172.18.0.2 -e DOCUMENT_SERVER_PORT_80_TCP_ADDR=documentserver -e DOCUMENT_SERVER_HOST=192.168.0.37 -v /app//onlyoffice/CommunityServer/data:/var/www/onlyoffice/Data -v /sharedfolders/share1/IT/docker/volumes/onlyoffice-logs/communityserver:/var/log/onlyoffice -v /app//onlyoffice/DocumentServer/data:/var/www/onlyoffice/DocumentServerData --env-file /srv/dev-disk-by-id-ata-TOSHIBA_HDWQ140_6769K0UBFPBE-part1/share1/nanda/development/sbs-tooling/Docker/onlyoffice/onlyoffice.env sbs/communityserver docker network connect onlyoffice onlyoffice --ip 172.18.0.6

I am attaching the /var/log/onlyoffice/Index/elasticsearch log. I see few errors there. It would be helpful if you could provide me pointers how to fix them.

elasticsearch.log

[Carazyda]

For what did you use --tmpfs labels, specifically /tmp? Seems errors happened because you mapped --tmpfs /tmp folder, it makes java crash "Could not initialize class com.sun.jna.Native". If you not set this parameter, elastic will works.

[loigu]

Thank you @Carazyda for the pointer. After deleting the --tmpfs labels elasticsearch started up. Yet fulltext search of documents & projects still does not work.

p.s. when I upgraded the container was not booting up properly - the --tmpfs labels were solution that I googled up and did make the container boot up little bit more. I assume they are not necessary when --privileged is specified.

[Carazyda]

Any new errors in logs? Service OnlyofficeIndex is working? Maybe it need some time to reindex your data.

[loigu]

@Carazyda : Service OnlyofficeIndex starts, but is always killed by watchdog after 1d

-- Unit onlyofficeIndex.service has finished starting up. -- -- The start-up result is RESULT. Dec 09 16:40:41 onlyoffice mono[9387]: /var/www/onlyoffice/Services/TeamLabSvc/TeamLabSvc.exe: Service started Dec 10 16:40:40 onlyoffice systemd[1]: onlyofficeIndex.service: Watchdog timeout (limit 1d)! Dec 10 16:40:40 onlyoffice systemd[1]: onlyofficeIndex.service: Killing process 9387 (mono) with signal SIGABRT. 0m Dec 10 16:40:42 onlyoffice systemd[1]: onlyofficeIndex.service: Main process exited, code=dumped, status=6/ABRT m Dec 10 16:40:42 onlyoffice systemd[1]: onlyofficeIndex.service: Failed with result 'watchdog'. Dec 10 16:40:48 onlyoffice systemd[1]: Starting Start ONLYOFFICE Index Service... -- Subject: Unit onlyofficeIndex.service has begun start-up -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit onlyofficeIndex.service has begun starting up. Dec 10 16:40:48 onlyoffice systemd[1]: Started Start ONLYOFFICE Index Service. -- Subject: Unit onlyofficeIndex.service has finished start-up -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- Unit onlyofficeIndex.service has finished starting up. -- -- The start-up result is RESULT. Dec 10 16:40:49 onlyoffice mono[19765]: /var/www/onlyoffice/Services/TeamLabSvc/TeamLabSvc.exe: Service started

no errors in Index/* svcIndex.log is full of

at System.Net.WebConnection.Connect (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x001e1] in <88f564ea69dd4dc8ba9bf979e48d5996>:0 --- End of inner exception stack trace --- at System.Net.WebConnection.Connect (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x0025d] in <88f564ea69dd4dc8ba9bf979e48d5996>:0 at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x000cc] in <88f564ea69dd4dc8ba9bf979e48d5996>:0 at System.Net.WebOperation.Run () [0x0009a] in <88f564ea69dd4dc8ba9bf979e48d5996>:0 at System.Net.WebCompletionSource1[T].WaitForCompletion () [0x00094] in <88f564ea69dd4dc8ba9bf979e48d5996>:0 at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <88f564ea69dd4dc8ba9bf979e48d5996>:0 at System.Net.HttpWebRequest.GetResponse () [0x00016] in <88f564ea69dd4dc8ba9bf979e48d5996>:0 at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse] (Elasticsearch.Net.RequestData requestData) [0x00082] in <651eba6aa786478b9ae0fef55b11a90b>:0 --- End of inner exception stack trace --- at Elasticsearch.Net.Transport1[TConnectionSettings].HandleElasticsearchClientException (Elasticsearch.Net.RequestData data, System.Exception clientException, Elasticsearch.Net.IElasticsearchResponse response) [0x00064] in <651eba6aa786478b9ae0fef55b11a90b>:0 at Elasticsearch.Net.Transport1[TConnectionSettings].FinalizeResponse[TResponse] (Elasticsearch.Net.RequestData requestData, Elasticsearch.Net.IRequestPipeline pipeline, System.Collections.Generic.List1[T] seenExceptions, TResponse response) [0x0004b] in <651eba6aa786478b9ae0fef55b11a90b>:0 at Elasticsearch.Net.Transport1[TConnectionSettings].Request[TResponse] (Elasticsearch.Net.HttpMethod method, System.String path, Elasticsearch.Net.PostData data, Elasticsearch.Net.IRequestParameters requestParameters) [0x00197] in <651eba6aa786478b9ae0fef55b11a90b>:0 at Elasticsearch.Net.ElasticLowLevelClient.DoRequest[TResponse] (Elasticsearch.Net.HttpMethod method, System.String path, Elasticsearch.Net.PostData data, Elasticsearch.Net.IRequestParameters requestParameters) [0x00006] in <651eba6aa786478b9ae0fef55b11a90b>:0 at Nest.ElasticClient.DoRequest[TRequest,TResponse] (TRequest p, Elasticsearch.Net.IRequestParameters parameters, System.Action1[T] forceConfiguration) [0x00076] in <197b0f8a5ba34ffa9c3749d16487721a>:0 at Nest.ElasticClient.Ping (Nest.IPingRequest request) [0x00008] in <197b0f8a5ba34ffa9c3749d16487721a>:0 at ASC.ElasticSearch.FactoryIndexer.CheckState (System.Boolean cacheState) [0x00057] in :0

[Carazyda]

Please, can you attach a full svcIndex.log?

[loigu]

@Carazyda svcIndex.log

[Carazyda]

In the log are old errors connection refused to elasticsearch, when it will been not running. Other Onlyoffice services are working well? You have enought ram on server?

[loigu]

Yes, other services seems to be running well. Mem usage is under 40% with 10GB of free ram.

[loigu]

@Carazyda Any other tip where I should look or any data/logs I can provide that would help you to discern where the problem might lie?

[Carazyda]

Please try running the latest 11.0.1 version of community server, this should help.

[loigu]

@Carazyda upgraded to "onlyoffice.community.version": "11.0.1.1482" elasticsearch and index starts, but still "doing nothing" (some exceptions in the logs).

there are some deprecation warnings/errors as well but I assume that should not break it completely. elasticsearch_deprecation.log svcIndex.log elasticsearch.log

[Carazyda]

You sure, that all services are running? svcIndex.log have only old errors of connection refused. Also maybe you need to reindex your data. Are you using our Control Panel?

[loigu]

@Carazyda Everything is reported as running. No I am not using Control Panel - if I am not mistaken it is not part of the "community/free" package. Correct? How do I force reindexing my data?

thank you Nanda

> 
> UNIT                                        LOAD   ACTIVE     SUB       DESCRIPTION                                            
> dev-sdc1.device                             loaded activating tentative /dev/sdc1                                              
> dev-sdd1.device                             loaded activating tentative /dev/sdd1                                              
> dev-sdd5.device                             loaded activating tentative /dev/sdd5                                              
> -.mount                                     loaded active     mounted   Root Mount                                             
> dev-mqueue.mount                            loaded active     mounted   POSIX Message Queue File System                        
> etc-hostname.mount                          loaded active     mounted   /etc/hostname                                          
> etc-hosts.mount                             loaded active     mounted   /etc/hosts                                             
> etc-letsencrypt.mount                       loaded active     mounted   /etc/letsencrypt                                       
> etc-resolv.conf.mount                       loaded active     mounted   /etc/resolv.conf                                       
> proc-sys-fs-binfmt_misc.mount               loaded active     mounted   Arbitrary Executable File Formats File System          
> var-lib-mysql.mount                         loaded active     mounted   /var/lib/mysql                                         
> var-log-onlyoffice.mount                    loaded active     mounted   /var/log/onlyoffice                                    
> var-www-onlyoffice-Data.mount               loaded active     mounted   /var/www/onlyoffice/Data                               
> var-www-onlyoffice-DocumentServerData.mount loaded active     mounted   /var/www/onlyoffice/DocumentServerData                 
> init.scope                                  loaded active     running   System and Service Manager                             
> binfmt-support.service                      loaded active     exited    Enable support for additional executable binary formats
> elasticsearch.service                       loaded active     running   Elasticsearch                                          
> god.service                                 loaded active     running   God Service                                            
> monoserve.service                           loaded active     running   FastCGI Mono server                                    
> monoserveApiSystem.service                  loaded active     running   FastCGI Mono server                                    
> nginx.service                               loaded active     running   nginx - high performance web server                    
> onlyofficeBackup.service                    loaded active     running   Start ONLYOFFICE Backup Service                        
> onlyofficeFeed.service                      loaded active     running   ONLYOFFICE Feed Service                                
> onlyofficeIndex.service                     loaded active     running   Start ONLYOFFICE Index Service                         
> onlyofficeJabber.service                    loaded active     running   ONLYOFFICE Jabber Service                              
> onlyofficeMailAggregator.service            loaded active     running   ONLYOFFICE MailAggregator Service                      
> onlyofficeMailCleaner.service               loaded active     running   ONLYOFFICE MailCleaner Service                         
> onlyofficeMailWatchdog.service              loaded active     running   ONLYOFFICE MailWatchdog Service                        
> onlyofficeNotify.service                    loaded active     running   ONLYOFFICE Notify Service                              
> onlyofficeRadicale.service                  loaded active     running   ONLYOFFICE Radicale Service                            
> onlyofficeSocketIO.service                  loaded active     running   ONLYOFFICE SocketIO Service                            
> onlyofficeStorageEncryption.service         loaded active     running   Start ONLYOFFICE StorageEncryption Service             
> onlyofficeStorageMigrate.service            loaded active     running   Start ONLYOFFICE StorageMigrate Service                
> onlyofficeTelegram.service                  loaded active     running   ONLYOFFICE Telegram Service                            
> onlyofficeThumb.service                     loaded active     running   ONLYOFFICE Thumb Service                               
> onlyofficeUrlShortener.service              loaded active     running   ONLYOFFICE UrlShortener Service                        
> redis-server.service                        loaded active     running   Advanced key-value store                               
> rsyslog.service                             loaded active     running   System Logging Service                                 
> systemd-journald.service                    loaded active     running   Journal Service                                        
> systemd-tmpfiles-setup-dev.service          loaded active     exited    Create Static Device Nodes in /dev                     
> systemd-tmpfiles-setup.service              loaded active     exited    Create Volatile Files and Directories                  
> -.slice                                     loaded active     active    Root Slice                                             
> system.slice                                loaded active     active    System Slice                                           
> dbus.socket                                 loaded active     listening D-Bus System Message Bus Socket                        
> syslog.socket                               loaded active     running   Syslog Socket                                          
> systemd-journald-audit.socket               loaded active     running   Journal Audit Socket                                   
> systemd-journald-dev-log.socket             loaded active     running   Journal Socket (/dev/log)                              
> systemd-journald.socket                     loaded active     running   Journal Socket                                         
> dev-sdd5.swap                               loaded active     active    /dev/sdd5                                              
> basic.target                                loaded active     active    Basic System                                           
> graphical.target                            loaded active     active    Graphical Interface                                    
> local-fs.target                             loaded active     active    Local File Systems                                     
> multi-user.target                           loaded active     active    Multi-User System                                      
> network-online.target                       loaded active     active    Network is Online                                      
> paths.target                                loaded active     active    Paths                                                  
> slices.target                               loaded active     active    Slices                                                 
> sockets.target                              loaded active     active    Sockets                                                
> swap.target                                 loaded active     active    Swap                                                   
> sysinit.target                              loaded active     active    System Initialization                                  
> timers.target                               loaded active     active    Timers                                                 
> certbot.timer                               loaded active     waiting   Run certbot twice daily                                
> systemd-tmpfiles-clean.timer                loaded active     waiting   Daily Cleanup of Temporary Directories  

[Carazyda]

In new version Control Panel is free, and distributed as a part of Onlyoffice Groups or Workspace. You can install and use it. In Control Panel we added a function to reindex data that you need. For manual reindex data you need enter in docker container of communityserver and run following commands: 1) Stop god and onlyofficeIndex services; 2) curl http://localhost:9200/_cat/indices for output available indices; 3) curl -X DELETE http://localhost:9200/name_index for delete selected index; 4) Clean webstudio_index table in onlyoffice database delete from webstudio_index;; 5) Start all services back.

[Carazyda]

This issue was closed due to no response.