Yadoms : retrieve huge data with charts

[jmberhault]

When retrieve some huge data, Yadoms crashed. No information in logs. I have to investigate the trouble.

[jmberhault]

The problem is in server side :

Error: signal 11: ./yadoms(_ZN6shared7process23CApplicationStopHandler12crashHandlerEi+0x18)[0x5690e8] /lib/arm-linux-gnueabihf/libc.so.6(__default_sa_restorer+0x0)[0x76bb76b0] ./yadoms(_ZN9rapidjson12GenericValueINS_4UTF8IcEENS_19MemoryPoolAllocatorINS_12CrtAllocatorEEEEC1IS5_EERKNS0_IS2_T_EERS5_b+0x1f8)[0x5bb2d0] ./yadoms(_ZN9rapidjson12GenericValueINS_4UTF8IcEENS_19MemoryPoolAllocatorINS_12CrtAllocatorEEEEC1IS5_EERKNS0_IS2_T_EERS5_b+0xc4)[0x5bb19c] ./yadoms(ZN6shared14CDataContainerC2ERKS0+0xcc)[0x57c164] ./yadoms(_ZNSt6vectorIN6shared14CDataContainerESaIS1_EE19_M_emplace_back_auxIJRKS1EEEvDpOT+0x84)[0x527c64] ./yadoms(_ZNK3web4rest7service12CAcquisition14getKeywordDataERKSt6vectorISsSaISsEERKSs+0x284)[0x48f6f0] ./yadoms(_ZN5boost6detail8function21function_obj_invoker2INS_3_bi6bind_tINS_10shared_ptrIN6shared13serialization17IDataSerializableEEENS_4_mfi4cmf2IS9_N3web4rest7service12CAcquisitionERKSt6vectorISsSaISsEERKSsEENS3_5list3INS3_5valueIPSF_EENS_3argILi1EEENSS_ILi2EEEEEEES9_SK_SM_E6invokeERNS1_15function_bufferESKSM+0x34)[0x4d4f6c] ./yadoms(_ZN3web4rest15CRestDispatcher14callRealMethodEN5boost9function2INS2_10shared_ptrIN6shared13serialization17IDataSerializableEEERKSt6vectorISsSaISsEERKSsEENS2_9function3IS8_SG_SD_SF_EESDSF+0x1c8)[0x3c4630] ./yadoms(ZN3web4rest15CRestDispatcher8dispatchERKSsRKSt6vectorISsSaISsEES3+0x254)[0x47b818]

[jano42]

The provided stacktrace do not illustrate HUGE data retreival.

It'is for getting data without entering in spceific "huge data management"

[sgallou]

Not reproduced with Yadoms 2.3.0-beta.3, using this REST request to display some various chart without problem

[sgallou]

Newly seen. Too much memory used. Need to work on chart and add security (data size limit) into server.

04:10:45 : : [Trace] : Rest request : [GET] : /rest/acquisition/keyword/468/20200219T031000/20200219T041045 04:10:45 : : [Trace] : Rest request : [GET] : /rest/acquisition/keyword/480/20200219T031000/20200219T041045 04:10:45 : : [Trace] : Rest request : [GET] : /rest/acquisition/keyword/1050/20200219T031000/20200219T041045 04:10:46 : : [Trace] : Rest request : [GET] : /rest/system/currentTime 04:10:46 : : [Trace] : Rest request : [GET] : /rest/acquisition/keyword/480/20200218T041000/20200219T041046 04:10:46 : : [Trace] : Rest request : [GET] : /rest/acquisition/keyword/468/20200218T041000/20200219T041046 04:10:47 : : [Trace] : Rest request : [GET] : /rest/acquisition/keyword/1050/20200218T041000/20200219T041046 04:10:50 : : [Trace] : Rest request : [GET] : /rest/eventLogger/from/1527 04:10:55 : : [Trace] : Rest request : [GET] : /rest/eventLogger/from/1527 04:11:00 : : [Debug] : DateTimeNotifier : schedule the task @04:12:00.000 04:11:00 : : [Trace] : Rest request : [GET] : /rest/eventLogger/from/1527 Error: signal 11: Error: signal 11: ./yadoms(_ZN6shared7process23CApplicationStopHandler12crashHandlerEi+0x18)[0x579330] /lib/arm-linux-gnueabihf/libc.so.6(__default_sa_restorer+0x0)[0xb6be46b0] ./yadoms(_ZN6shared7process23CApplicationStopHandler12crashHandlerEi+0x18)[0x579330] /lib/arm-linux-gnueabihf/libc.so.6(__default_sa_restorer+0x0)[0xb6be46b0] ./yadoms./yadoms(_ZN9rapidjson12GenericValueINS_4UTF8IcEENS_19MemoryPoolAllocatorINS_12CrtAllocatorEEEEC1IS5_EERKNS0_IS2_T_EERS5_b+0x1f8)[0x5cebe4] (_ZN9rapidjson12GenericValueINS_4UTF8IcEENS_19MemoryPoolAllocatorINS_12CrtAllocatorEEEEC1IS5_EERKNS0_IS2_T_EERS5_b+0x1f8)[0x5cebe4] ./yadoms(_ZN9rapidjson12GenericValueINS_4UTF8IcEENS_19MemoryPoolAllocatorINS_12CrtAllocatorEEEEC1IS5_EERKNS0_IS2_T_EERS5_b+0xc4)[0x5ceab0] ./yadoms(_ZN9rapidjson12GenericValueINS_4UTF8IcEENS_19MemoryPoolAllocatorINS_12CrtAllocatorEEEEC1IS5_EERKNS0_IS2_T_EERS5_b+0xc4)[0x5ceab0] ./yadoms(ZN6shared14CDataContainerC2ERKS0+0xcc)[0x589044] ./yadoms./yadoms(ZN6shared14CDataContainerC2ERKS0+0xcc)[0x589044] (_ZNSt6vectorIN6shared14CDataContainerESaIS1_EE19_M_emplace_back_auxIJRKS1EEEvDpOT+0x84)[0x537338] ./yadoms(_ZNSt6vectorIN6shared14CDataContainerESaIS1_EE19_M_emplace_back_auxIJRKS1EEEvDpOT+0x84)[0x537338] ./yadoms./yadoms(_ZNK3web4rest7service12CAcquisition14getKeywordDataERKSt6vectorISsSaISsEERKSs+0x270)[0x496118] (_ZNK3web4rest7service12CAcquisition14getKeywordDataERKSt6vectorISsSaISsEERKSs+0x270)[0x496118] ./yadoms(_ZN5boost6detail8function21function_obj_invoker2INS_3_bi6bind_tINS_10shared_ptrIN6shared13serialization17IDataSerializableEEENS_4_mfi4cmf2IS9_N3web4rest7service12CAcquisitionERKSt6vectorISsSaISsEERKSsEENS3_5list3INS3_5valueIPSF_EENS_3argILi1EEENSS_ILi2EEEEEEES9_SK_SM_E6invokeERNS1_15function_bufferESKSM+0x34)[0x4ddd30] ./yadoms./yadoms(_ZN5boost6detail8function21function_obj_invoker2INS_3_bi6bind_tINS_10shared_ptrIN6shared13serialization17IDataSerializableEEENS_4_mfi4cmf2IS9_N3web4rest7service12CAcquisitionERKSt6vectorISsSaISsEERKSsEENS3_5list3INS3_5valueIPSF_EENS_3argILi1EEENSS_ILi2EEEEEEES9_SK_SM_E6invokeERNS1_15function_bufferESKSM+0x34)[0x4ddd30] (_ZN3web4rest15CRestDispatcher14callRealMethodEN5boost9function2INS2_10shared_ptrIN6shared13serialization17IDataSerializableEEERKSt6vectorISsSaISsEERKSsEENS2_9function3IS8_SG_SD_SF_EESDSF+0x1c4)[0x3ca798] ./yadoms(_ZN3web4rest15CRestDispatcher14callRealMethodEN5boost9function2INS2_10shared_ptrIN6shared13serialization17IDataSerializableEEERKSt6vectorISsSaISsEERKSsEENS2_9function3IS8_SG_SD_SF_EESDSF+0x1c4)[0x3ca798] ./yadoms./yadoms(ZN3web4rest15CRestDispatcher8dispatchERKSsRKSt6vectorISsSaISsEES3+0x26c)[0x481ab8] (ZN3web4rest15CRestDispatcher8dispatchERKSsRKSt6vectorISsSaISsEES3+0x26c)[0x481ab8]

[sgallou]

@jmberhault Chart widget must not request so much data. Server side, I will do :

• Accept request on raw data ONLY if time duration specified is less than 2 hours
• Limit database request to 2000 records (add a log warning entry if limit reached)

Chart side, what must be done :

• Remove "all" button for "day" period
• Rework on requests

What we have to do :

• Discuss about a solution to give pertinent data between raw and hourly summaries.

[jano42]

The issue has been reproduced by writing units tests !!

This is a strange memory allocation behavior. I'm working on fixing it

[jano42]

Bug identified precisely. This is rapidjson memory allocation and using CDataContainer in std::vector which produce the fault.

I'm working on enhancements:

• try to limit (or prohibit) use of std::vector<shared::CDatacontainer> (to do)
• add new CDataContainer methods to handle arrays (done)
• tweak rapidjson to make its memory management better adapted to yadoms (benchmark in progress, to determine best parameter set)

[jano42]

After lot of tests, here a first conclusion.

• tweaking rapidjson library is not a good solution. After tests it appears that actual configuration is the best.
• we can also keep lib as it is and provide a default buffer, so the use may tell the container how many space are needed (with automatic reallocation if needed): shared::CDataContainer myContainer(56); //56 preallocated bytes myContainer.set("a", 12); //will use allocated buffer myContainer.set("b", 24); //will use allocated buffer .... myContainer.set("verylongdata", "Ideoque fertur neminem aliquando ob hae[....]"); //will use allocate 65k more buffers on the heap
• make the copy constructor private (so use of CDataContainer is very restricted, so heap allocation is better controlled, and data duplication limited) => rapidjson already forbids copies. => lot of work, everywhere CDataContainer are copied, then need to change it to shared_ptr (really good solution)
• fix the webserver methods which request too much data, by limiting amount and data, and fixing the way data is retreived and stored before reply to client)

Realizing all those updates will:

• make yadoms faster (avoid data copy, this is theorical, not really measurable)
• limit the memory allocation (avoid the huge data exception)
• yadoms will surely be more robust (!?!?)

[jano42]

Fixed in develop branch