ADS - Retention - Dispose job failed for Unstructured Data

[suseela22]

Retention - Disposition is not working for unstructured data.

Applied Retention via Job Management (Table level) for Records containing Blobs. Generate purge job was successful. While running dispose job, an error stating that the path is not a "valid DFS filename"

Screenshots

Additional context Build no : 2024.03.45414_P1 Region : http://192.168.1.218:4081/ Client :

[kilam005]

@Nadarajan-suyambu / @ashoksri0 can one of you point the Root casue? I see the issue fixed, but can you also infom ETA as to when it will be packaged for testing and delivery.

[Nadarajan-suyambu]

RCA: Blob path appended two times (/lwa/warehouse/fugro_saudi_dbo/unstructure_year_blobs/fugro_saudi_dbo/unstructure_year_blobs)

FixGiven: Blob path construction happened in both Java and Scala sides, instead of this the base path only forwards to the Scala side. Screenshots:

[Nadarajan-suyambu]

This issue was resolved in Tag '2024.09.45542_RC1'.

[Udhayanila8]

_Server: http://192.168.56.146 Version: 2024.09.45542RC1 Unable to validate this issue due to retention set is not creating. Create record level retention set. Apply expired retention set for record-level The retention set is not creating, a job is in progress for a long time, so couldn't check this issue.

[Udhayanila8]

_Server: 192.168.56.146 Version: 2024.09.45542RC1

If the application is under S3 storage but the table is under HDFS storage dispose purge job is failed.

Purge List for unstructured data Run the dispose purge item Application is s3 storage but the table under is HDFS storage Purge dispose got failed. But change the application storage from s3 to HDFS Now trigger the dispose job, It completed successfully.

[kilam005]

@ashoksri0 to check and get back with details of the issue and fix details

[harsha-kotha]

Add Root cause details for associated issue identified by Nila during testing.

[kilam005]

@ClementJosh21 to update details.

[maheswarareddyi]

@Udhayanila8 Please verify and provide the status of the fix.

[kilam005]

testing has not been initiated. Planned to complete by today.

[Udhayanila8]

Server : 192.168.56.146 Version: 2024.10.45564_RC3

Issue identified in the retention creation from job management page. Reference link: https://github.com/P3-Core-Dev-Team/ADS_BE/issues/3248

Due to that issue unable to retest this issue, so currently holding this.

[kilam005]

An updated build is provided today. @Udhayanila8 please start checking again.

[Udhayanila8]

Server: 192.168.56.146 Version: 2024.10.45564_RC4

The depended issue still not closed, job is in-progress for a long time. https://github.com/P3-Core-Dev-Team/ADS_BE/issues/3248

[kilam005]

@Udhayanila8 to check and provide updates

[Udhayanila8]

_Server: 192.168.56.146 Version: 2024.10.45564RC6

Unable to retest this issue, due to dependent issue, Application is under s3 storage Changed the table storage as hdfs After ingestion record count While creating the retention policy in the job management page, the following issue was raised. https://github.com/P3-Core-Dev-Team/ADS_BE/issues/3381 Due to that, retesting is blocked

[kilam005]

PR is awaiting merge to proceed with fix delivery for this issue. @dinesh-p3 To validate and retest and then pass it for testing. ETA for PR merge and test completion : EOW