augustocristian
commented
3 months ago I attempted to fix the issue yesterday in this branch. I tried the following approaches:
-
Modify the SQL database health check: I attempted to query the database for the existence of the five tables. However, this approach was not effective because the various APIs, which are waiting for the database to be healthy, are responsible for creating the data and model. This led to a deadlock in the SUT, as the first service was waiting for a state that was never going to occur.
-
Check the status prior to executing the test cases: This was not useful because the system failure occurs beforehand. Although the test cases would run (with the database having the necessary tables), the involved services would not respond due the failure
-
Create a custom health check in the order-api service: I spent most of my time on this approach. My idea was to modify the order-api Dockerfile as follows:
# Custom health check for RETORCH parallelization # Copy the health check script COPY "Services/Ordering/Ordering.API/checklogs.sh" "/usr/local/bin/checklogs.sh" # Make the script executable RUN chmod +x "/usr/local/bin/checklogs.sh"In the checklogs script, I tried to grep the message:
#!/bin/bash if docker logs $HOSTNAME 2>&1 | grep -q "Now listening on: http://0.0.0.0:80"; then exit 0 else exit 1 fiHowever, I encountered the following problems:
- The message is printed to the container's stdout and can be accessed with the
docker logscommand. I considered allowing the container to access the Docker socket and execute Docker commands, but I discarded this idea due to security reasons. - I looked for a file (e.g., in /var/logs) containing the stdout content without success.
- After discarding the first two options, I tried modifying the service to configure its logging system to also log information messages to a file by changing the appsettings.json as follows:
{ "Logging": { "LogLevel": { "Default": "Information", "Microsoft": "Warning", "Microsoft.Hosting.Lifetime": "Information" }, "File": { "Path": "/logs/myapp.log", "LogLevel": { "Default": "Information" } } } }However, this approach was also unsuccessful.
- The message is printed to the container's stdout and can be accessed with the
javiertuya
In a previous pull request (PR), we experienced intermittent failures (approximately 1 out of every 15 executions) due to a 502 Bad Gateway error. I investigated the root cause of this issue, i the URLPR, I increased the timeout for the migration process, which partially addresses the problem. However, to enhance the stability of the System Under Test (SUT), we need a more maintainable solution. The normal execution flow includes logging the following information in the order-api:
Now listening on: http://0.0.0.0:80This indicates that the Microsoft.eShopOnContainers.Service.OrdersDB database has been created and populated with data. However, depending on the system load, there can be service desynchronization, causing the Backend for Frontends (BFF) to experience failures and report incorrect redirections. The failures are reported as migration timeout exceptions, as shown in the logs below:The database error log also shows that it is not ready:
For more details, please refer to the following logs: