search

fledge-iot / fledge

An open source platform for the Industrial Internet of Things, it acts as an edge gateway between sensor devices and cloud storage systems.
https://www.lfedge.org/projects/fledge/
Apache License 2.0
121 stars 44 forks source link

Error at restart with a high number of services #1315

Open geofjamg opened 6 months ago

geofjamg commented 6 months ago

I have 150 south services and 10 north services and when I stop then start fledge I have the following error:

Mar 20 10:10:48 machine1 Fledge [1990997] INFO: script.fledge: Fledge started.
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.common.configuration_manager: Unable to create new category based on category_name iec104_s99 and category_description iec104_s99 an
d category_json_schema {'plugin': {'description': 'iec104 south plugin', 'type': 'string', 'readonly': 'true', 'default': 'iec104', 'value': 'iec104'}, 'asset': {'description': 'Asset name', 'type':
 'string', 'order': '1', 'displayName': 'Asset Name', 'mandatory': 'true', 'default': 'iec104', 'value': 'iec104'}, 'protocol_stack': {'description': 'protocol stack parameters', 'type': 'JSON', 'or
der': '2', 'displayName': 'Protocol stack parameters', 'default': '{ "protocol_stack" : { "name" : "iec104client", "version" : "1.0", "transport_layer" : { "redundancy_groups" : [ { "connections" : 
[ { "srv_ip" : "127.0.0.1", "port" : 2404 } ], "rg_name" : "red-group1", "tls" : false, "k_value" : 12, "w_value" : 8, "t0_timeout" : 10, "t1_timeout" : 15, "t2_timeout" : 10, "t3_timeout" : 20 } ] 
}, "application_layer" : { "orig_addr" : 10, "ca_asdu_size" : 2, "ioaddr_size" : 3, "asdu_size" : 0, "gi_time" : 60, "gi_cycle" : 60, "gi_all_ca" : false, "utc_time" : false, "cmd_wttag" : false, "c
md_parallel" : 0, "time_sync" : 0 } } }', 'value': '{ "protocol_stack" : { "name" : "iec104client", "version" : "1.0", "transport_layer" : { "redundancy_groups" : [ { "connections" : [ { "srv_ip" : 
"127.0.0.1", "port" : 2404 } ], "rg_name" : "red-group1", "tls" : false, "k_value" : 12, "w_value" : 8, "t0_timeout" : 10, "t1_timeout" : 15, "t2_timeout" : 10, "t3_timeout" : 20 } ] }, "application
_layer" : { "orig_addr" : 10, "ca_asdu_size" : 2, "ioaddr_size" : 3, "asdu_size" : 0, "gi_time" : 60, "gi_cycle" : 60, "gi_all_ca" : false, "utc_time" : false, "cmd_wttag" : false, "cmd_parallel" : 
0, "time_sync" : 0 } } }'}, 'exchanged_data': {'description': 'exchanged data list', 'type': 'JSON', 'order': '3', 'displayName': 'Exchanged data list', 'default': '{ "exchanged_data": { "name" : "i
ec104client", "version" : "1.0", "datapoints" : [ { "label":"TM-1", "protocols":[ { "name":"iec104", "address":"41025-4202832", "typeid":"M_ME_NA_1" } ] }, { "label":"TM-2", "protocols":[ { "name":"
iec104", "address":"41025-4202852", "typeid":"M_ME_NA_1" } ] }, { "label":"TS-1", "protocols":[ { "name":"iec104", "address":"41025-4206948", "typeid":"M_SP_TB_1" } ] } ] } }', 'value': '{ "exchange
d_data": { "name" : "iec104client", "version" : "1.0", "datapoints" : [ { "label":"TM-1", "protocols":[ { "name":"iec104", "address":"41025-4202832", "typeid":"M_ME_NA_1" } ] }, { "label":"TM-2", "p
rotocols":[ { "name":"iec104", "address":"41025-4202852", "typeid":"M_ME_NA_1" } ] }, { "label":"TS-1", "protocols":[ { "name":"iec104", "address":"41025-4206948", "typeid":"M_SP_TB_1" } ] } ] } }'}
, 'tls': {'description': 'tls parameters', 'type': 'JSON', 'order': '4', 'displayName': 'TLS parameters', 'default': '{ "tls_conf" : { "private_key" : "iec104_client.key", "own_cert" : "iec104_clien
t.cer", "ca_certs" : [ { "cert_file": "iec104_ca.cer" }, { "cert_file": "iec104_ca2.cer" } ], "remote_certs" : [ { "cert_file": "iec104_server.cer" } ] } }', 'value': '{ "tls_conf" : { "private_key"
 : "iec104_client.key", "own_cert" : "iec104_client.cer", "ca_certs" : [ { "cert_file": "iec104_ca.cer" }, { "cert_file": "iec104_ca2.cer" } ], "remote_certs" : [ { "cert_file": "iec104_server.cer" 
} ] } }'}}
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration: Failed to create category.
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration: Traceback (most recent call last):
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/sources/fledge/python/fledge/services/core/api/configuration.py"
, line 149, in create_category#012    await cf_mgr.create_category(category_name=category_name, category_description=category_desc,
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/sources/fledge/python/fledge/common/configuration_manager.py", l
ine 1280, in create_category#012    all_categories = await self._read_all_category_names()
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/sources/fledge/python/fledge/common/configuration_manager.py", l
ine 616, in _read_all_category_names#012    results = await self._storage.query_tbl_with_payload('configuration', payload)
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/sources/fledge/python/fledge/common/storage_client/storage_clien
t.py", line 308, in query_tbl_with_payload#012    jdoc = await resp.json()
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/.local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", li
ne 1098, in json#012    await self.read()
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/.local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", li
ne 1036, in read#012    self._body = await self.content.read()
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/.local/lib/python3.8/site-packages/aiohttp/streams.py", line 375
, in read#012    block = await self.readany()
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/.local/lib/python3.8/site-packages/aiohttp/streams.py", line 397
, in readany#012    await self._wait("readany")
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration:   File "/home/gj/.local/lib/python3.8/site-packages/aiohttp/streams.py", line 304
, in _wait#012    await waiter
Mar 20 10:11:31 machine1 Fledge[1991148] ERROR: logger: fledge.services.core.api.configuration: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Mar 20 10:11:31 machine1 Fledge iec104_s99[1994560]: ERROR: Failed to add configuration category: Response payload is not completed.

I also have sometimes this kind of crash:

Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: Signal 6 (Aborted) trapped:
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (0) 0       0x5561e7f6e9b2 handler(int) + 82---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (1) 1       0x7f55bff59090 /lib/x86_64-linux-gnu/libc.so.6(+0x43090) [0x7f55bff59090]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (2) 2       0x7f55bff5900b gsignal + 203---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (3) 3       0x7f55bff38859 abort + 299---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (4) 4       0x7f55c01c38d1 /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e8d1) [0x7f55c01c38d1]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (5) 5       0x7f55c01cf37c /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa37c) [0x7f55c01cf37c]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (6) 6       0x7f55c01cf3e7 /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3e7) [0x7f55c01cf3e7]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (7) 7       0x7f55c01cf699 /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa699) [0x7f55c01cf699]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (8) 8       0x7f55c068d9ec /home/gj/sources/fledge/cmake_build/C/lib/libcommon-lib.so.1(+0x539ec) [0x7f55c068d9ec]---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (9) 9       0x5561e7f74890 SouthService::createConfigCategories(DefaultConfigCategory, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 208---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (10) 10      0x5561e7f74e06 SouthService::loadPlugin() + 582---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (11) 11      0x5561e7f757d6 SouthService::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, unsigned short) + 1014---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (12) 12      0x5561e7f5d8c7 main + 631---------
Mar 19 13:54:18 machine1 Fledge iec104_s3[752002]: FATAL: (13) 13      0x7f55bff3a083 __libc_start_main + 243---------
Mar 19 13:54:18 machine1 Fledge[720832] ERROR: logger: fledge.services.core.api.configuration: Failed to create the child relationship for South category.

Any idea what could be the issue and if there is a way to workaround ?

MarkRiddoch commented 5 months ago

I suspect this is all triggered by the services starting together and overloading the API of the core service. We have done the first state of adding the ability to order the start up, which will help with this. But we need to probably complete that work south that we can stagger the startup of the south services. Or maybe even have a mechanism that automatically rate limits the service startup when we start getting large numbers.

geofjamg commented 5 months ago

I tried with just one south service configured and when I stop then restart fledge I get more than 300 hundreds http requests across all services just for them to restart (there is non data acquisition). I excluded ping http requests.

POST /fledge/service HTTP/1.1
GET /fledge/service/category/Storage HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/Advanced/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
POST /fledge/interest HTTP/1.1
..POST /fledge/interest HTTP/1.1
.PUT /storage/reading/query HTTP/1.1
PUT /storage/table/streams/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /storage/table/scheduled_processes HTTP/1.1
GET /storage/table/schedules HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/ping HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
.GET /fledge/ping HTTP/1.1
POST /fledge/interest HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/statistics/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/Storage/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
POST /fledge/interest HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/Storage/children HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
POST /fledge/interest HTTP/1.1
PUT /fledge/service/category/Storage/plugin HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /fledge/service/category/Storage/readingPlugin HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/asset_tracker/query HTTP/1.1
PUT /storage/table/alerts/query HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /storage/table/log HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/Utilities/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
GET /fledge/service/category/purge_system HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
GET /fledge/ping HTTP/1.1
PUT /storage/table/statistics/query HTTP/1.1
Fledge started.
GET /fledge/service/ping HTTP/1.1
GET /fledge/service/ping HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category/iec104_s1 HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/South/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category/iec104_s1 HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/iec104_s1/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category/iec104_s1Advanced HTTP/1.1
POST /fledge/service HTTP/1.1
POST /storage/table/log HTTP/1.1
POST /fledge/interest HTTP/1.1
POST /fledge/interest HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/schema/fledge/table/statistics/query HTTP/1.1
GET /fledge/service/category/iec104_s1 HTTP/1.1
GET /fledge/service/category/iec104_pivot_filter HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/iec104_s1%20Filters/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
POST /fledge/service/category/iec104_s1/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category/iec104_s1_iec104_pivot_filter HTTP/1.1
POST /fledge/service/category/iec104_s1/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/interest HTTP/1.1
POST /fledge/service/category?keep_original_items=true HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
POST /fledge/service/category/iec104_s1/children HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
PUT /storage/table/category_children/query HTTP/1.1
PUT /storage/table/configuration/query HTTP/1.1
GET /fledge/service/category/iec104_s1Security HTTP/1.1
POST /fledge/interest HTTP/1.1
POST /storage/reading HTTP/1.1
PUT /storage/schema/fledge/table/statistics HTTP/1.1
POST /storage/table/tasks HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/schema/fledge/table/statistics/query HTTP/1.1
POST /storage/schema/fledge/table/statistics_history HTTP/1.1
PUT /storage/schema/fledge/table/statistics HTTP/1.1
PUT /storage/table/tasks HTTP/1.1
POST /storage/table/tasks HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/schema/fledge/table/statistics/query HTTP/1.1
POST /storage/schema/fledge/table/statistics_history HTTP/1.1
PUT /storage/schema/fledge/table/statistics HTTP/1.1
PUT /storage/table/tasks HTTP/1.1
POST /storage/table/tasks HTTP/1.1
GET /fledge/service?name=Fledge%20Storage HTTP/1.1
PUT /storage/schema/fledge/table/statistics/query HTTP/1.1
POST /storage/schema/fledge/table/statistics_history HTTP/1.1
PUT /storage/schema/fledge/table/statistics HTTP/1.1
PUT /storage/table/tasks HTTP/1.1
geofjamg commented 5 months ago

@MarkRiddoch I found the issue. Simple-Web-Server has a default request timeout of 5s: image This is a little bit short when restarting 160 services because the storage server is flooded of request during a few seconds. I tried to increase it and all my services finally restart without any error. Would it be ok for you to be able to configure in fledge the the timeout_request and the timeout_content so that we can adapt it better to use cases?

geofjamg commented 5 months ago

The hack I did to make it working: image

MarkRiddoch commented 5 months ago

A new configuration item has been added to the Advanced - Storage category that allows this timeout to be set. The default has also been increased. This is in the develop branch and will be in tonights nightly build.