upload_frame() showing random behavior

[j-honnacker]

I have flat files with timestamp data that can be correctly loaded into CAS tables by using the e8601dt19. informat in the importOptions of the loadTable action (SAS Studio + PROC CAS).

However, I noticed that this informat produces random results when used in the upload_frame() method of the SWAT package. Here is an example:

# Creating dataFrame
df = pd.DataFrame({
    'Date1' : pd.date_range(start='2023-12-15 10:00:00', end='2024-01-15 10:00:00', freq='D'),
    'Date2' : pd.date_range(start='2023-12-15 10:00:00', end='2024-01-15 10:00:00', freq='D')
})

When I now execute the following part multiple times, the dataframe is sometimes uploaded correctly, sometimes it contains missing values in one or both columns:

# Uploading dataFrame to CAS and viewing table
test = conn.upload_frame(
    df,
    importoptions=dict(
    vars = [
            { 'name' : 'Date1', 'type' : 'datetime', 'informat' : 'e8601dt19.'},
            { 'name' : 'Date2', 'type' : 'datetime', 'informat' : 'e8601dt19.'}
        ]
    ),
    casout={
        'caslib':'casuser',
        'name':'test',
        'replace':True
    }
)

test.head(20)

Possible outcome with 3 missing values:

Viya 2023.12 Python 3.11.7 pandas 2.1.4 swat 1.13.3

[j-honnacker]

NVM, the bug is with the Viya loadTable action, not with the SWAT package.

Just learned that we apply the informat within a DATA step AFTER execution of loadTable (where it would lead to random missings).