E.g., initially opening the 20GB Europe_glcp_extended.csv from edi.394.5 in prod took around 3 1/2 minutes total, and the processing spinner was showing only during the last half or so, of that time.
The issue is probably that the CSV related cache entries are created at the same time as the EML based ones.
Todo:
Consider disconnecting the creation of EML based cache entries from the CSV related ones.
Consider adding some sort of loading screen.
Check to make sure that DeX in prod is finding the CSVs in the NFS mount. The download path for objects in PASTA should only kick in for staging and development. Even if reading from the full CSV from NFS mount is no faster than downloading it, reading from NFS will be faster when only part of the CSV is to be loaded (since the full CSV has to be downloaded even when only part of it is to be loaded).
E.g., initially opening the 20GB
Europe_glcp_extended.csvfromedi.394.5in prod took around 3 1/2 minutes total, and the processing spinner was showing only during the last half or so, of that time.The issue is probably that the CSV related cache entries are created at the same time as the EML based ones.
Todo:
Consider disconnecting the creation of EML based cache entries from the CSV related ones.
Consider adding some sort of loading screen.
Check to make sure that DeX in prod is finding the CSVs in the NFS mount. The download path for objects in PASTA should only kick in for staging and development. Even if reading from the full CSV from NFS mount is no faster than downloading it, reading from NFS will be faster when only part of the CSV is to be loaded (since the full CSV has to be downloaded even when only part of it is to be loaded).