Client Error '400 Bad Request' on performing any command output to falcon syncer

N-0867 commented 1 year ago

First Stop

[X] I have read and agreed to the ThoughtSpot Dev Tools EULA.
[X] I understand that if I have a critical issue with ThoughtSpot, I should reach out to my designated Account Executive and Solutions Consultant team instead of cs_tools.
[X] I searched the cs_tools documentation, with the integrated search.

Platform Configuration

│ Info snapshot taken on 2023-08-14 │ │ │ │ CS Tools: 1.4.9 │ │ Python Version: Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] │ │ System Info: Windows (detail: Windows-10-10.0.19045-SP0) │ │ Configs Directory: C:\Users\ {anonymous} \AppData\Roaming\cs_tools │ │ Activate VirtualEnv: C:\Users\ {anonymous} \AppData\Roaming\cs_tools.cs_tools\Scripts\Activate.ps1 │ │ Platform Tags: win-amd64

Description

Hi, I have been a user of cs_tools for a while now, and I have just gone to run a searchable gather to output to an instance, which I have the config and syncer file created and was working fine before. This has resulted with an error of "Client Error '400 bad Request'" immediately on the stage " Writing Groups Privileges to falcon".

The command I am running is; cs_tools tools searchable gather --config D1 --syncer falcon://default

Has something changed with the setup of CS_tools in the latest version, which has been missed (most likely by myself!) or in the documentation? The only change we have had recently is the setup of our instances - which have moved to AL2 and as EBS-backed storage. Any help would be appreciated!

boonhapus commented 1 year ago

@Hey @N-0867 !

Would you be able to re-run the command and then share your latest log file?

Please run.. cs_tools logs report , save it to a location you prefer, and add it to the issue or load it to Egnyte (if you feel you don't want to expose some of that data).

N-0867 commented 1 year ago

Hi @boonhapus, I have extracted the error from the logs pasted below - removing some references to IDs etc.

Not sure if there is something I am missing from the below, but I can see the falcon database an schema exists (the data tables are created within ThoughtSpot when the cs_tools searchable command is run).

`Error 400: Bad Request Details: error code = INVALID_ARGUMENT, message = Cannot commit data load for cycle_id=[XXXX] as it is in stage=[cycle_id: "XXXX" start_time: 1692087238653574 ingested_network_bw: 0 buffered_bytes_size: 0 rows_written: 0 bytes_written: 0 ignored_row_count: 0 ]. [Level 5 - 2023-08-15 09:13:59,870] [cs_tools.api._rest_api_v1 - _rest_api_v1.request 116] >> HEADERS:

Headers({'host': 'server-ip:XXXX', 'content-length': '0', 'accept': '/', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'x-requested-by': 'CS Tools', 'user-agent': 'cs_tools/1.4.9 (+github: thoughtspot/cs_tools)', 'cookie': 'JSESSIONID=XXXX; clientId=XXXX'}) [DEBUG - 2023-08-15 09:13:59,924] [cs_tools.api._rest_api_v1 - _rest_api_v1.request 84] >> POST to callosum/v1/tspublic/v1/session/logout with keywords {} [DEBUG - 2023-08-15 09:13:59,968] [cs_tools.api._rest_api_v1 - _rest_api_v1.request 109] << HTTP: 204 [DEBUG - 2023-08-15 09:13:59,970] [cs_tools.cli.main - main.run 169] whoopsie, something went wrong! Traceback (most recent call last): File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\cs_tools\cli\main.py", line 144, in run return_code = app(standalone_mode=False) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\typer\main.py", line 328, in call raise e File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\typer\main.py", line 311, in call return get_command(self)(*args, kwargs) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\click\core.py", line 1130, in call return self.main(args, kwargs) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\typer\core.py", line 778, in main return _main( File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\typer\core.py", line 216, in _main rv = self.invoke(ctx) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\click\core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\click\core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\click\core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\cs_tools\cli\ux.py", line 38, in invoke r = ctx.invoke(self.callback, ctx.params) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\click\core.py", line 760, in invoke return __callback(args, kwargs) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\typer\main.py", line 683, in wrapper return callback(use_params) # type: ignore File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\cs_tools\cli\tools\searchable\app.py", line 300, in gather syncer.dump("ts_group_privilege", data=data) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\cs_tools\cli\layout.py", line 113, in exit raise exc File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\cs_tools\cli\tools\searchable\app.py", line 300, in gather syncer.dump("ts_group_privilege", data=data) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\cs_tools\sync\falcon\syncer.py", line 107, in dump self.ts.tsload.upload( File "pydantic\decorator.py", line 40, in pydantic.decorator.validate_arguments.validate.wrapper_function File "pydantic\decorator.py", line 134, in pydantic.decorator.ValidatedFunction.call File "pydantic\decorator.py", line 206, in pydantic.decorator.ValidatedFunction.execute File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\cs_tools\api\middlewares\tsload.py", line 300, in upload self.ts.api.dataservice_dataload_commit(cycle_id=data["cycle_id"]) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\cs_tools\api_rest_api_v1.py", line 656, in dataservice_dataload_commit r = self.post(fullpath) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\cs_tools\api_rest_api_v1.py", line 66, in post return self.request("POST", endpoint, passthru) File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\cs_tools\api_rest_api_v1.py", line 117, in request r.raise_for_status() File "C:\local-file-reference\cs_tools.cs_tools\lib\site-packages\httpx_models.py", line 749, in raise_for_status raise HTTPStatusError(message, request=request, response=self) httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://server-ip:XXXX/ts_dataservice/v1/public/loads/XXXX/commit' For more information check: https://httpstatuses.com/400 `

boonhapus commented 1 year ago

ack, thank you @N-0867, this is helpful direction. let me try to reproduce internally.

boonhapus commented 5 months ago

Falcon Sync should be fixed as part of 1.5.0 !

thoughtspot / cs_tools