Structure of FM parameters is cached

[Pentusha]

Hello. I've got functional module ZHHDEMO_STRUCT_MOD with EX_ZHHT_COL2 param.

FUNCTION ZHHDEMO_STRUCT_MOD.
*"----------------------------------------------------------------------
*"*"Local Interface:
*"  EXPORTING
*"     VALUE(EX_ZHHT_COL2) TYPE  ZHHTT_COL2
*"----------------------------------------------------------------------

  DATA: ls_col2 TYPE zhhs_col2.

  ls_col2-COL1 = 'col1_1'.
  ls_col2-COL2 = 'col2_1'.

  APPEND ls_col2 TO EX_ZHHT_COL2.

  ls_col2-COL1 = 'col1_2'.
  ls_col2-COL2 = 'col2_2'.

  APPEND ls_col2 TO EX_ZHHT_COL2.

ENDFUNCTION.

And I've got python script that performs the following sequence of actions:

1. Prints structure of EX_ZHHT_COL2 parameter;
2. Prints function result ZHHDEMO_STRUCT_MOD;
3. Gives me the time to modify structure of EX_ZHHT_COL2 parameter;
4. Prints structure of EX_ZHHT_COL2 param again. This is proof that the structure has been changed;
5. Prints function result ZHHDEMO_STRUCT_MOD which exactly the same as pt 2.

import pyrfc

connection_info = {
    # some connection data here
}

function_name = u'ZHHDEMO_STRUCT_MOD'
table_name = u'ZHHTT_COL2'

def get_structure():
    with pyrfc.Connection(**connection_info) as con:
        interface_response = con.call(
            'RFC_GET_FUNCTION_INTERFACE',
            **{'FUNCNAME': function_name}
        )
        assert any(p[u'TABNAME'] == table_name for p in interface_response[u'PARAMS'])
        structure_response = con.call(
            'RFC_GET_STRUCTURE_DEFINITION',
            **{'TABNAME': table_name}
        )
        fields = structure_response[u'FIELDS']
        return [f[u'FIELDNAME'] for f in fields]

def function_call():
    with pyrfc.Connection(**connection_info) as con:
        return con.call(function_name)

if __name__ == '__main__':
    print('STRUCTURE 1', get_structure())
    print('RESULT 1', function_call())
    raw_input('Structure changed now. Press Enter to continue...')
    print('STRUCTURE 2', get_structure())
    print('RESULT 2', function_call())

I change the structure as follows:

Python script gives me output:

('STRUCTURE 1', [u'ZHHS_COL2', u'COL1', u'COL2'])
('RESULT 1', {u'EX_ZHHT_COL2': [{u'COL2': u'col2_1', u'COL1': u'col1_1'}, {u'COL2': u'col2_2', u'COL1': u'col1_2'}]})
Structure changed now. Press Enter to continue...
('STRUCTURE 2', [u'ZHHS_COL2', u'COL1', u'COL2', u'COL3'])
('RESULT 2', {u'EX_ZHHT_COL2': [{u'COL2': u'col2_1', u'COL1': u'col1_1'}, {u'COL2': u'col2_2', u'COL1': u'col1_2'}]})

The next launch gives me the right output:

('STRUCTURE 1', [u'ZHHS_COL2', u'COL1', u'COL2', u'COL3'])
('RESULT 1', {u'EX_ZHHT_COL2': [{u'COL2': u'col2_1', u'COL3': u'', u'COL1': u'col1_1'}, {u'COL2': u'col2_2', u'COL3': u'', u'COL1': u'col1_2'}]})
Structure changed now. Press Enter to continue...
('STRUCTURE 2', [u'ZHHS_COL2', u'COL1', u'COL2', u'COL3'])
('RESULT 2', {u'EX_ZHHT_COL2': [{u'COL2': u'col2_1', u'COL3': u'', u'COL1': u'col1_1'}, {u'COL2': u'col2_2', u'COL3': u'', u'COL1': u'col1_2'}]})

[bsrdjan]

That is actually correct behaviour. The pyrfc caches the RFC metadata to gain performance and does not support changing the RFC signature during runtime. Is there a real problem/requirement and scenario in which RFC parameters' structures change during runtime ?

[Pentusha]

I've got a daemon/webserver which periodically call function. So I need to restart it when structure is changed. Is there a way to invalidate cache?

[bsrdjan]

There is a nwrfc lib api available to invalidate the cache, for a given sysid and function module name. Would it be enough to expose it as pyrfc method? It should be called before calling the RFC. Would that be ok?

[Pentusha]

Yes, it will be ok.

[bsrdjan]

The fix is available in dev branch now, also the ubuntu wheel. Could you please check if works for you? Method names and examples how to use are in respective unit test.

Please consider that if incorrect function module name supplied, like 'XXXXXX_STRUCT_MOD' for example, no error is returned because not finding it in cache is not an error. The removal from cache does not check if the function module exists. If such check needed, the application must take care.

[Pentusha]

I added a deletion of the cache before function call and recompile pyrfc with your commit (in new virtual environment), but unfortunately I still get same result.

def function_call():
    with pyrfc.Connection(**connection_info) as con:
        con.func_desc_remove(connection_info['sysid'], function_name)
        response = con.call(function_name)
        return response

('STRUCTURE 1', [u'ZHHS_COL2', u'COL1', u'COL2'])
('RESULT 1', {u'EX_ZHHT_COL2': [{u'COL2': u'col2_1', u'COL1': u'col1_1'}, {u'COL2': u'col2_2', u'COL1': u'col1_2'}]})
Structure changed now. Press Enter to continue...
('STRUCTURE 2', [u'ZHHS_COL2', u'COL1', u'COL2', u'COL3'])
('RESULT 2', {u'EX_ZHHT_COL2': [{u'COL2': u'col2_1', u'COL1': u'col1_1'}, {u'COL2': u'col2_2', u'COL1': u'col1_2'}]})

[bsrdjan]

after more investigation I am sorry to confirm the feature can't be supported and cache invalidation can't help. The DDIC structure changes are visible only in the same session. If the change is done by another session in backend, the pyrfc session can't see it, only after logout/login. That is application server restriction, not of nwrfclib or pyrfc.

[Pentusha]

Thanks for the explanation. I reproduced it in the SAP GUI. Can you explain me what do the terms "session", "login/logout" stands for. I thought that the opening of the connection is equal to the login and the start of the session, and closing of connection is equal to logout and drop the session. Why session drops when I terminate process that uses sapnwrfc? I might be wrong.

[bsrdjan]

"login/logout" terms are used in relation to sapgui client, used to run transactions like SE11, SE80 ... Login to sapgui client creates a user session, just like opening a pyrfc connection. On SAP Help more info are available.

As your test environment is already configured, could you please try one more thing, to call the reset_server_context() method, instead of removing from cache?

[Pentusha]

reset_server_context() does not help.

[bsrdjan]

ok, that was the last hope. sorry that I have to confirm that the feature is not supported. The structure changes can be visible only after closing and re-opening the connection. The question will not be forgotten and if any other solution comes in the meantime, I will update this thread.

[Pentusha]

I see, but still cannot understood which connection I should close and re-open. SAP GUI or pyrfc? Each pyrfc request is made in a new connection. SAP GUI restarting has no matter too. Even reimport of pyrfc has no matter, just process termination.

[bsrdjan]

The pyrfc connection. To recap, the test case is following:

1. Open the pyrfc connection and read the RFC structure
2. Change the RFC structure in backend (SAP GUI), while pyrfc connection still open
3. Re-read the RFC structure by pyrfc, without re-starting python script or re-opening the connection.

The change is here not visible, this is the current behaviour and the problem, correct?

Reading the initial description, the next launch of the python script (process termination) shows the change, correct ?

My expectation was that the change shall be visible after closing and opening the connection, without re-starting the script. If that not the case, need to test locally.

[Pentusha]

Changes are not visible in different connections until restarting the script.

Pseudocode:

connction.open()
function_call()
# structure changed with sap gui
function_call()  # changes are not visible here
connection.close()

connection.open()
function_call()  # changes are not visible even here
connection.close()

[bsrdjan]

Could you please do one more test, with the latest dev branch ?

I tested with the structure export parameter and following invalidation worked for me:

con.func_desc_remove(sys_id, function_name)
con.type_desc_remove(sys_id, struct_name)

The sys_id is 3 characters system id, of the backend system. Function and struct names for your test should be:

con.func_desc_remove(sys_id, 'ZHHDEMO_STRUCT_MOD')
con.type_desc_remove(sys_id, 'ZHHTT_COL2')
con.type_desc_remove(sys_id, 'ZHHS_COL2')

[Pentusha]

con.func_desc_remove(connection_info['sysid'], 'ZHHDEMO_STRUCT_MOD')
con.type_desc_remove(connection_info['sysid'], 'ZHHTT_COL2')
con.type_desc_remove(connection_info['sysid'], 'ZHHS_COL2')
con.reset_server_context()
con.call(...)

did not help too

[bsrdjan]

That is strange. I replicated with exactly the same names of data elements and RFC module and the result is positive. Could you please try with this test script ? Opening new connections is not necessarily required, reset_server_context also not.

connection_info = {
    'user': '',
    'passwd': '',
    'ashost': '',
    'saprouter': '',
    'sysnr': '',
    'lang': '',
    'client': '',
    'sysid': ''
}

import pyrfc

function_name = u'ZHHDEMO_STRUCT_MOD'
table_name = u'ZHHT_COL2'
struct_name = u'ZHHS_COL2'
field_name = u'COL3'

def get_structure():
    with pyrfc.Connection(**connection_info) as con:
        interface_response = con.call(
            'RFC_GET_FUNCTION_INTERFACE',
            **{'FUNCNAME': function_name}
        )
        assert any(p[u'TABNAME'] == table_name for p in interface_response[u'PARAMS'])
        structure_response = con.call(
            'RFC_GET_STRUCTURE_DEFINITION',
            **{'TABNAME': table_name}
        )
        fields = structure_response[u'FIELDS']
        return [f[u'FIELDNAME'] for f in fields]

def function_call():
    with pyrfc.Connection(**connection_info) as con:
        return con.call(function_name)

def invalidate():
    with pyrfc.Connection(**connection_info) as con:
        con.func_desc_remove(connection_info['sysid'], function_name)
        con.type_desc_remove(connection_info['sysid'], table_name)
        con.type_desc_remove(connection_info['sysid'], struct_name)
        #con.reset_server_context()

if __name__ == '__main__':
    print('STRUCTURE 1', get_structure())
    print('RESULT 1', function_call())
    raw_input('Structure changed now. Press Enter to continue...')
    invalidate()
    print('STRUCTURE 2', get_structure())
    print('RESULT 2', function_call())

Here the console output of my test:

(dev)$ python issue40a.py
('STRUCTURE 1', [u'ZHHS_COL2', u'COL1', u'COL2'])
('RESULT 1', {u'EX_ZHHT_COL2': [{u'COL2': u'col2_1', u'COL1': u'col1_1'}]})
Structure changed now. Press Enter to continue...
('STRUCTURE 2', [u'ZHHS_COL2', u'COL1', u'COL2', u'COL3'])
('RESULT 2', {u'EX_ZHHT_COL2': [{u'COL2': u'col2_1', u'COL3': u'', u'COL1': u'col1_1'}]})

[Pentusha]

I just updated the nwrfcsdk library to latest version and got the same results. And I did a new research with interesting results: https://gist.github.com/Pentusha/2039485a583781e7dd30f99b5a753e2c After changing the structure and restarting the script, the cache already stores the correct value at the first access. It means structure is already cached even before function call and it not changed even after cache invalidation. Does this mean that the structure description keeps into the cache on the first connection?

[bsrdjan]

Sorry, I am not surre I fully follow the idea. The gist script does not make a break, to wait until the data element changed in backend. What is the purpose of the test and what kind of functionality is exactly requested?

Backend data elements metadata are cached in nwrfc lib and shared among connections. The cache can be invalidated by above mentioned methods (or process restart).

Does that solve the initial problem ?

The get_cache method does not read from cache. It calls the nwrfc lib api RfcGetTypeDesc for example, requesting the metadata. The cache will be checked first and if not in cache, will be read from backend. Hope that explains the output of the script.

[Pentusha]

I figured out this problem. The fact is that the sysid is case insensitive when connected, but it is case sensitive while cache invalidation.