Integration tests executed on a real deployment as part of the CICD - Datasets

[dlpzx]

Same as for https://github.com/data-dot-all/dataall/issues/1220.

This issue is to track the progress for the Datasets modules. It has its own dedicated issue because of the challenge of pre-existing infrastructure needed to test datasets

[dlpzx]

Required tests for basic coverage

https://github.com/data-dot-all/dataall/pull/1379

For fresh deployments

For each of the following API calls we need to test authorized and unauthorized scenarios as well as all possible configurations (e.g. autoapproval...)

• [X] Create Dataset - includes testing if dataset is indexed in Catalog
• [X] Import Dataset - includes testing if dataset is indexed in Catalog
• [X] List Datasets
• [x] Edit Dataset
• [x] Delete dataset
• [x] Start crawler
• [x] Sync tables
• [x] Preview table
• [x] Delete table
• [x] Create folder
• [x] Delete folder

For backwards compatibility

• [x] Update created Dataset stack
• [x] Update imported Dataset stack
• [x] Create Dataset in updated Environment
• [x] Import Dataset in updated Environment

Full coverage

For fresh deployments

• [x] All Table and Folder APIs: https://github.com/data-dot-all/dataall/pull/1391
• [x] All Table profiling jobs APIs: https://github.com/data-dot-all/dataall/pull/1533
• [x] All Table columns API calls: https://github.com/data-dot-all/dataall/pull/1548
• [x] All Table data filter API calls: https://github.com/data-dot-all/dataall/pull/1539

For backwards compatibility

For the updated Dataset stacks:

• [ ] AWS access to Dataset - Credentials
• [ ] AWS access to Dataset - S3 redirect
• [ ] Start crawler
• [ ] Sync tables