E_ABORT - Githubissues

alex4o commented 5 years ago

When reporting a new issue include the following information:

[6.0.0 ] sfctl version, can check this using pip show sfctl
[ 6.4.625, latest from docker hub ] Service Fabric cluster runtime version

[ docker sfonebox ] Service Fabric hosting location (on-premise or Azure usually)


DEBUG: Command arguments: ['application', 'upload', '--path', 'data-test1', '--show-progress', '--debug']
DEBUG: Event: Cli.PreExecute []
DEBUG: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fd1dabeda60>, <function OutputProducer.on_global_arguments at 0x7fd1dab8e840>, <function CLIQuery.on_global_arguments at 0x7fd1dabb5bf8>]
DEBUG: Event: CommandInvoker.OnPreCommandTableCreate [<function CLILogging.remove_logger_flags at 0x7fd1dabedae8>]
DEBUG: Event: CommandLoader.OnLoadArguments []
DEBUG: Event: CommandInvoker.OnPostCommandTableCreate []
DEBUG: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: Event: CommandInvoker.OnPreParseArgs []
DEBUG: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fd1dab8e8c8>, <function CLIQuery.handle_query_parameter at 0x7fd1dabb5c80>]
DEBUG: urllib3.connectionpool : Starting new HTTP connection (1): 127.0.0.1:19080
DEBUG: urllib3.connectionpool : http://127.0.0.1:19080 "PUT /ImageStore/data-test1/ApplicationManifest.xml?api-version=6.1 HTTP/1.1" 503 60
ERROR: 503 Server Error: E_ABORT for url: http://127.0.0.1:19080/ImageStore/data-test1/ApplicationManifest.xml?api-version=6.1
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/knack/cli.py", line 125, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib/python3.7/site-packages/knack/invocation.py", line 85, in execute
cmd_result = parsed_args.func(params)
File "/usr/lib/python3.7/site-packages/knack/commands.py", line 67, in __call__
return self.handler(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/knack/commands.py", line 123, in _command_handler
result = op(client, **command_args) if client else op(**command_args)
File "/usr/lib/python3.7/site-packages/sfctl/custom_app.py", line 154, in upload
show_progress)
File "/usr/lib/python3.7/site-packages/sfctl/custom_app.py", line 102, in upload_to_native_imagestore
res.raise_for_status()
File "/usr/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: E_ABORT for url: http://127.0.0.1:19080/ImageStore/data-test1/ApplicationManifest.xml?api-version=6.1


Debug log if an error is present (can be captured using `--debug` flag)
I tried everything to upload but it does not work. I don't know what to do anymore.

Christina-Kang commented 5 years ago

Hi @alex4o,

Thank you for reporting this issue. To help us diagnose, can you send us the traces from your local cluster and the approximate time (including timezone) of when you ran the upload command? You can email me the traces at bikang@microsoft.com, or you can upload them on here. They are located at /mnt/sfuser/log/lttng-traces or /mnt/sfroot. The files should be either .etl or .trace.

Thanks!

alex4o commented 5 years ago

I hope these helps. I have used the TraceProcessor.sh to get the trace file. The time should be around 16:30 utc. I m not sure that this repo is the right place to report, because I also tried doing this:

# curl "http://127.0.0.1:19080/ImageStore?api-version=6.0"                                        ~/Projects/scynet/sf/test-dotnet/data-test1
{"Error":{"Code":"FABRIC_E_NAME_DOES_NOT_EXIST","Message":"Name does not exist."}}

So it is possible that it is a problem with service fabric itself. trace.zip

Christina-Kang commented 5 years ago

@alex4o can you try running the following in order?

Sudo userdel P_FSSUserffffffff
Sudo userdel S_FSSUserffffffff
groupdel FSSGroupffffffff
Sudo systemctl restart servicefabric

Give this a shot, and if doesn't work (for both the upload and the GET), could you share the traces again? Thanks!

alex4o commented 5 years ago

I tried the commands that you have suggested, but as I am doing this in a docker container I don't have systemd(systemctl), so i just ran the the ./run.sh provided by the official docker image. It did not work. traces.7z.gz

Christina-Kang commented 5 years ago

@alex4o Let's set up a call to diagnose this. Can you send me a email to the address given above? Thanks!

microsoft / service-fabric-cli

E_ABORT #165