search

vanderby / SonarQube-AzureAppService

Instructions and files to host SonarQube on an Azure App Service without a container.
MIT License
90 stars 165 forks source link

SonarQube flood stage disk watermark [95%] exceeded on #40

Closed inoreip closed 4 years ago

inoreip commented 4 years ago

I am raising this issue here as it is specific to the way Azure App Service allocates disk space on the D drive and D:/home folder. I have not been able to work around this and there are 1 or 2 posts online with the same issue with no robust resolution.

WARN es[][o.e.c.r.a.DiskThresholdMonitor] flood stage disk watermark [95%] exceeded on [0WrMWG4XSEObU4chJQUUvg][sonarqube][D:\home\site\wwwroot\sonarqube-7.9\data\es6\nodes\0] free: 1.2gb[3.9%], all indices on this node will be marked read-only

https://github.com/elastic/elasticsearch/issues/53233

https://github.com/vanderby/SonarQube-AzureAppService/issues/35

The above post mentioned scaling up the App Service Plan but this doesn't work if your index is approaching 30GB and is marked as Closed.

The Elasticsearch index in SonarQube is set to a 95% disk threshold meaning that when used disk space is going to exceed 95% then Elasticsearch writes a warning in the es.log file and tries to write to another node. However there are no other nodes so the indexing enters an infinite loop. In Elasticsearch you can tweak these settings in the elasticsearch.yml file, e.g. you can turn this check off. However SonarQube seems to ignore this yml file and creates a temporary one on startup in a temp folder. I tried adding cluster.routing.allocation.disk.threshold_enabled: false to the yml file which was ignored on App Service restart. I did this to prevent Elasticsearch from checking disk space as I have allocated way more than enough in the App Service Plan.

The more canny might state well just add more disk space (if only it was that straight forward). Scaling up an App Service Plan adds more disk space to D:/home but not D:/ which seems to be the OS disk and is fixed around 32GB. D:/home is what looks like a symbol link or share which is actually a dedicated disk from the App Service Plan. I have the plan at 250GB which I can see in Kudu (Azure environment tool). It looks like Elasticsearch is using the free disk space on D: and not the folder where the index is stored e.g. D:/home/site/wwwroot/.../data.

I suspect this issue cannot be solved in Azure, SonarQube doesn't honour Elasticsearch settings and Elasticsearch just uses the Java libraries to check free disk space. What I am trying to say is I don't think any of the 3 would take ownership and solve this problem, rendering the App Service deployment only fit for small deployments with small indexes. This is a shame as in my opinion this deployment is the most manageable and robust option.

If anybody is able to tell me otherwise I would be delighted to see a resolution. Thank you.

vanderby commented 4 years ago

Thanks for the detailed analysis. I agree none of the 3 would take ownership directly. So it's on a motivated individual from this community to submit an issue & PR to SQ for a fix.

If we can adjust this by adding a setting to sonar.properties that gets passed to ES, here's an open PR we could use as an example

Rincey commented 4 years ago

Just want to add a 'me too' We solved it (this time) by going from S1 to P1 and thus getting another 200GB of disk space.... But the original 50GB (49.3GB free), to my way of thinking, should be HEAPS!!!

Would love a solution to this too.....

vanderby commented 4 years ago

Adding the ES documentation page for config settings that are relevant to this discussion. https://www.elastic.co/guide/en/elasticsearch/reference/6.8/disk-allocator.html

inoreip commented 4 years ago

Just want to add a 'me too' We solved it (this time) by going from S1 to P1 and thus getting another 200GB of disk space.... But the original 50GB (49.3GB free), to my way of thinking, should be HEAPS!!!

Would love a solution to this too.....

I would be surprised if this actually works after a few days. I did exactly the same going to a P1 instance. Its not the disk size of the app service that is the problem. It is that Microsoft use a symbol link to attach a drive under the App Service VMs D: drive which as far as I can tell is 32GB no matter what plan you choose. In Elasticsearch the Java File System library is used to check the root drive free disk space which shows 32GB give or take. If it checked the actual folder that Elasticsearch index was stored in it would see the full 50GB or 250GB depending on what plan you are on in the App Service. I am reverting to hosting SonarQube through an Azure Windows VM via IIS reverse proxy for now.

vanderby commented 4 years ago

I figured out a way to disable the disk check. Changing this setting did persist across App Service restarts.

  1. Enable ES HTTP Port by setting sonar.search.httpPort in the app settings.

    Notes:

    1. I set the port to 9200 since that is the ES default.
    2. SQ will log a warning not to enable this for production. But since the port is firewalled from the internet we should be safe here.
    3. The App Service should restart on its own after updating the app setting, but if not manually restart it.
  2. Open the Kudu PowerShell debug console "https://[web-app-name].scm.azurewebsites.net/DebugConsole/?shell=powershell" . Run the two commands below. 
    $ProgressPreference = "SilentlyContinue" # This disables the progress meter from accessing the console which is not allowed when executing on an App Service
Invoke-WebRequest -Uri http://localhost:9200/_cluster/settings -ContentType 'application/json' -Method PUT -Body '{ "persistent": { "cluster.routing.allocation.disk.threshold_enabled":false }}' -UseBasicParsing
  3. Verify by the various means:
    1. ES.log should say 
      INFO  es[][o.e.c.s.ClusterSettings] updating [cluster.routing.allocation.disk.threshold_enabled] from [true] to [false]
    2. Query ES Http API 
      Invoke-WebRequest -Uri http://localhost:9200/_cluster/settings -Method GET -UseBasicParsing
StatusCode        : 200
StatusDescription : OK
Content           : {"persistent":{"cluster":{"routing":{"allocation":{"disk":{ "threshold_enabled":"false"}}}}},"transient":{}}
...

Lastly, ES does log this warning about http.enabled being deprecated. Once SQ updates to a newer version of ES these steps may need to be reworked. From ES.log:

WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.

Reference For The Future:

inoreip commented 4 years ago

I figured out a way to disable the disk check. Changing this setting did persist across App Service restarts.

  1. Enable ES HTTP Port by setting sonar.search.httpPort in the app settings.

    Notes:

    1. I set the port to 9200 since that is the ES default.
    2. SQ will log a warning not to enable this for production. But since the port is firewalled from the internet we should be safe here.
    3. The App Service should restart on its own after updating the app setting, but if not manually restart it.
  2. Open the Kudu PowerShell debug console "https://[web-app-name].scm.azurewebsites.net/DebugConsole/?shell=powershell" . Run the two commands below. 
    $ProgressPreference = "SilentlyContinue" # This disables the progress meter from accessing the console which is not allowed when executing on an App Service
Invoke-WebRequest -Uri http://localhost:9200/_cluster/settings -ContentType 'application/json' -Method PUT -Body '{ "persistent": { "cluster.routing.allocation.disk.threshold_enabled":false }}' -UseBasicParsing

  3. Verify by the various means:

    1. ES.log should say 
      INFO  es[][o.e.c.s.ClusterSettings] updating [cluster.routing.allocation.disk.threshold_enabled] from [true] to [false]
    2. Query ES Http API 
      Invoke-WebRequest -Uri http://localhost:9200/_cluster/settings -Method GET -UseBasicParsing
StatusCode        : 200
StatusDescription : OK
Content           : {"persistent":{"cluster":{"routing":{"allocation":{"disk":{ "threshold_enabled":"false"}}}}},"transient":{}}
...

Lastly, ES does log this warning about http.enabled being deprecated. Once SQ updates to a newer version of ES these steps may need to be reworked. From ES.log:

WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.

Reference For The Future:

Excellent I couldn't get the ES API to work from the App Service and the port was the missing element I didn't configure. I would be interested to see where the setting is persisted to in ES for it to be resilient to App Service restarts.

@vanderby do you think it would be worth adding this to the PowerShell script wrapper as an option so the API call is made on startup to verify the disk allocation check? I'm thinking this could help others who will come across the same problem and also would remove the need to manually call the API in Kudo. I appreciate the setting is disabled in future versions but for now it is the only working solution and can be looked at when SQ and ES is updated.

Thank you.

vanderby commented 4 years ago

@inoreip, that is next on my list when I get time. It will be a bit more tricky since we can't communicate with the ES service until it starts. My initial thoughts are to spin up a separate PowerShell job that will poll for the ES service to start then make the web request to change the setting. Adding the setting to the app service to enable ES http will be easy.

In the meantime, verify if this solves your problem and let me know if it does not.

inoreip commented 4 years ago

@vanderby I've been trying to get the PowerShell commands to successfully execute via the Kudu interface. Unfortunately I stumbled upon the protocol validation issue with Invoke-WebRequest so I got around this using this article https://blog.toddlichty.com/using-powershell-to-check-a-file-full-of-urls/. After getting passed this error I was unable to connect to localhost:9200 even though ES logged that it listening on this port. I therefore tried Test-NetConnection localhost -port 9200 which worked sometimes and not others. It seems either ES is restarting a lot or the web request command executed on the App Service is a bit flaky.

Have you managed to successfully call the ES API via Kudu PowerShell?

vanderby commented 4 years ago

I ran into the protocol violation too when I was trying to send a request to the ES endpoint that was not the HTTP endpoint.

After setting sonar.search.httpPort in appsettings did you verify SQ restarted? You should see a message like this in ES.log that verifies the port the HTTP endpoint is listening on.

Elasticsearch HTTP connector is enabled on port {}. MUST NOT BE USED FOR PRODUCTION"

Kudu PowerShell was the only thing I was using when I was working with this issue.

inoreip commented 4 years ago

@vanderby My fault I set sonar.search.port which was commented out in the file instead of sonar.search.httpPort. I can confirm that by setting the correct property the API call succeeded and the index completed building. I am able to login and access all the functions of SonarQube now.

I can confirm it also handled an App Service Stop and Start. I could see in the es.log file that the setting was changed from enabled to disabled so I didn't have to call the API again.

Thank you

Rincey commented 4 years ago

Interesting.... Thanks for the hard work. I'll get this implemented for us too... I know I'm going to be asked, but whats the risk having port 9200 open? We don't use any IP restrictions or private vNets for our SQ webapp.

Rincey commented 4 years ago

I should add I agree with @inoreip - where is this being stored in ES? If not elastic.yaml then where? Because would it be easier set the setting in there, rather than open a port and execute a REST call.

Rincey commented 4 years ago

Thought: Once the ES settings have been set, can we close off the listener again?

vanderby commented 4 years ago

@Rincey I have no idea the ramifications of leaving the ES http endpoint open. Part of me says to not worry about it since Azure Web Apps blocks all ports to the open internet except 443. From a security aspect though you may as well turn it off if it's not needed.

As far as I can tell once you set this setting you can disable the http endpoint. I don't know where ES stores this setting so I won't guess if upgrading to different SQ version won't reset it.

BigA442 commented 4 years ago

Thanks so much. We had encountered this issue a few weeks ago, tried several things and ended up re installing after breaking something trying to fix it. When it came back we found this issue, have been trying things and following this. Once we saw the work around we implemented it, and no issues.

This is a great way for us to get up and running quickly using SonaQube.

zemien commented 3 years ago

Adding some newer context since I've just gone through this with SonarQube 8.9.2, which is bundled with (newer) ElasticSearch 7.

Updated steps from @vanderby original guide:

  1. Open the Kudu PowerShell debug console "https://[web-app-name].scm.azurewebsites.net/DebugConsole/?shell=powershell". Run the two commands below: 
    $ProgressPreference = "SilentlyContinue" # This disables the progress meter from accessing the console which is not allowed when executing on an App Service
Invoke-WebRequest -Uri http://localhost:9001/_cluster/settings -ContentType 'application/json' -Method PUT -Body '{ "persistent": { "cluster.routing.allocation.disk.threshold_enabled":false }}' -UseBasicParsing
  2. Verify ES.log should say: 
    INFO  es[][o.e.c.s.ClusterSettings] updating [cluster.routing.allocation.disk.threshold_enabled] from [true] to [false]
  3. Restart the App Service (optional but recommended).

Relevant notes from the ES upgrade ticket SONAR-12686:

One thing that wasn't discussed was whether this workaround needs to be applied with each SonarQube upgrade. I'm guessing so since each version contains its own distribution of ElasticSearch. Hence, I'm making a note to always run this command after every upgrade.

phydya commented 2 years ago

Hi, although this post is about SonarQube on Azure, I had the same problem with SonarQube on MacOS. And the solution given by @zemien was ok : curl -X PUT -H 'Content-Type: application/json' -d '{ "persistent": { "cluster.routing.allocation.disk.threshold_enabled":false }}' http://localhost:9001/_cluster/settings (tested on SonarQube 9.2.3) Thanks 🥇