Hash tab support refactoring

[sbcgua]

follows #85, also should solve #88 quite a big refactoring ... plus lots of new UTs

Architecture considerations

• In order to properly insert a line into hashed/sorted table it's key must be populated first
• we can fill the fill table line and insert it after. But it will hold a copy of the line in memory before insertion and we don't know what is the in the line ( structure of tables of tables ... ). Which is presumably memory inefficient (this was the choice at the end)
• we can also fill key fields (values only allowed) first, insert the line and then fill it in a usual way. But ! Json fields are case sensitive (may be in any case) and abap fields are not. So we cannot read specific fields of a structure directly. We have to loop through the whole structure in search for the key fields and then loop again for all others. Alternatively we can add a upper-cased name to the json tree, but it will add processing time for all cases including those, without hashed table. Which is unacceptable imho.
• Thus, the current approach is do double-processing of hashed/sorted tab records which makes it twice slower than usual. Some optimisations are possible. And also there is always a possibility to convert json into standard table and then assign it to hashed. So currently I think it is a fair alternative

[sbcgua]

Hmmm .... (this is the impact on NON-hashed tables code :( ) Kind of 50% slower ... :(((((

AJSON simple performance test
BEFORE
PLAIN_OBJ                      rounds: 1000,        result: 0.2069760
DEEP_OBJ                       rounds: 1000,        result: 0.3454670
ARRAY                          rounds: 1000,        result: 0.7944140
BIG_AND_COMPLEX                rounds: 10,          result: 14.0475390

AFTER
PLAIN_OBJ                      rounds: 1000         result: 0.3056790
DEEP_OBJ                       rounds: 1000         result: 0.4865090
ARRAY                          rounds: 1000         result: 1.0320220
BIG_AND_COMPLEX                rounds: 10           result: 16.0086790

[larshp]

get it working first, then make it fast 🤓

[sbcgua]

It kind of worked for me before ... and was fast. That what I complain about :)

P.S. it works now by the way. all UTs passes, though some minor tweaks to be done and maybe some performance studies/updates. I'm unhappy at the moment because what I do use became slower without any benefits for me :) (except the satisfaction of implementing a relatively complex problem)

[sbcgua]

BTW @albertmink , does the code from this branch works for you ?

[larshp]

try running SE30 and check what takes the most time?

[sbcgua]

yes, I will, but later. But I anticipate what I will see there - "any_to_abap takes most of the time". But it is a large method and so it will not be clear what to optimize. And splitting it into more methods will slow down it a bit more. But, maybe I'm wrong, I'll look into the profiler, I just need to rewrite the perf test a bit, it is not properly separates the tests

[sbcgua]

prepare_long_array ? really ? string operations are that slow in abap, or I'm missing something ? :))) And yeah, any_to_abap takes time ... for ... something inside it :)))

[sbcgua]

By the way I restructured perf tests ... in fact the convertion for a larger objects (which is probably the reality) takes less ... so maybe not that bad :)

before

TO_ABAP_PLAIN_OBJ              rounds: 1000         result: 0.0814990
TO_ABAP_DEEP_OBJ               rounds: 1000         result: 0.1889880
TO_ABAP_ARRAY                  rounds: 1000         result: 0.5362250
TO_ABAP_LONG_ARRAY             rounds: 5            result: 7.3524580
TO_ABAP_COMPLEX                rounds: 5            result: 5.4112050

after

TO_ABAP_PLAIN_OBJ              rounds: 1000         result: 0.1153050
TO_ABAP_DEEP_OBJ               rounds: 1000         result: 0.1888680
TO_ABAP_ARRAY                  rounds: 1000         result: 0.3385480
TO_ABAP_LONG_ARRAY             rounds: 5            result: 3.4077510
TO_ABAP_COMPLEX                rounds: 5            result: 2.2680440

[sbcgua]

... and changing cl_abap_typedescr=>typekind_** to just h or u reduced a couple of milliseconds by the way. Hmmm. But where is the rest of the time ? 3 seconds for 5 rounds of 5000 lines array ? Really ? Java script laughs out loudly and rolls on the floor ...

[sbcgua]

before

TO_ABAP_PLAIN_OBJ              rounds: 1000         result: 0.0354270
TO_ABAP_DEEP_OBJ               rounds: 1000         result: 0.0781470
TO_ABAP_ARRAY                  rounds: 1000         result: 0.2100300
TO_ABAP_LONG_ARRAY             rounds: 5            result: 2.7832750
TO_ABAP_COMPLEX                rounds: 5            result: 2.1166440

after

TO_ABAP_PLAIN_OBJ              rounds: 1000         result: 0.0455000
TO_ABAP_DEEP_OBJ               rounds: 1000         result: 0.0837050
TO_ABAP_ARRAY                  rounds: 1000         result: 0.1399000
TO_ABAP_LONG_ARRAY             rounds: 5            result: 1.3851790
TO_ABAP_COMPLEX                rounds: 5            result: 0.9504610

That's all I can do for the moment.

@larshp @albertmink - please check if this code fits your needs and I'll merge. That was fun to work on it, but I have to do other staff :)

[albertmink]

Happy to see the progress in numbers🙌 I can't wait to test it, maybe I find some time next week

[albertmink]

It works for my initial use-case. Thanks a lot

[larshp]

merge?

[sbcgua]

Yeap. Merging. Hold tight :) Hopefully unit tests will save us from bugz :) @albertmink thanks for the original issue and test case