automated curation lab fails (containers, 2.8)

planetf1 commented 3 years ago

When the 'automated curation' lab notebook is run in the kubernetes environment, it fails at the following cell:

oakDeneLandingAreaFullPath = "/" + landingAreaDirectoryName + "/oak-dene/clinical-trials/drop-foot"

os.listdir (oakDeneLandingAreaFullPath)

with

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-48-6a9a15d261f8> in <module>
      1 oakDeneLandingAreaFullPath = "/" + landingAreaDirectoryName + "/oak-dene/clinical-trials/drop-foot"
      2 
----> 3 os.listdir (oakDeneLandingAreaFullPath)

FileNotFoundError: [Errno 2] No such file or directory: '/deployments/data/landing-area/hospitals/oak-dene/clinical-trials/drop-foot'

This is failing as os.listdir is being used, which will list files locally - ie on the notebook server, and not on the actual egeria server.

The issue is minor - everything else has worked correctly, and so for 2.8 I propose adding to release notes, and considering how we want to do this properly in the next release

planetf1 commented 3 years ago

(setting to 2.8 until I've updated release notes)

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

mandy-chessell commented 3 years ago

This is still work to do

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

planetf1 commented 2 years ago

I took another look at it when testing 3.6 -- this isn't a regression to be concerned with for the release, but does seem to be failing more than it used to:

Step 15 (get integration status) fails with a bad directory name data/data-lake/.....


getIntegrationDaemonStatus(exchangeDL01Name, exchangeDL01PlatformName, exchangeDL01PlatformURL, petersUserId)

One integration service defined for integration daemon exchangeDL01

Integration Service: Files Integrator OMIS Integration Connector Reports: Integration Connector: OakDeneLandingAreaFilesMonitor status: RUNNING lastStatusChange: 2022-02-25T19:16:21.808+00:00 lastRefreshTime: 2022-02-25T19:16:22.291+00:00 minMinutesBetweenRefresh: 10 Integration Connector: OldMarketLandingAreaFilesMonitor status: FAILED failingExceptionMessage: BASIC-FILES-INTEGRATION-CONNECTORS-404-001 The directory named data/landing-area/hospitals/old-market/clinical-trials/drop-foot does not exist Integration Connector: DropFootClinicalTrialResultsFolderMonitor status: FAILED failingExceptionMessage: BASIC-FILES-INTEGRATION-CONNECTORS-404-001 The directory named data/data-lake/research/clinical-trials/drop-foot/weekly-measurements does not exist

 * step 21 then fails -- with a 404/not found on the call to configuration properties

Response: { "timestamp": "2022-02-25T19:16:32.432+00:00", "status": 404, "error": "Not Found", "path": "/servers/exchangeDL01/open-metadata/integration-daemon//users/peterprofile/integration-services/files-integrator/connectors/configuration-properties" }

Response: { "timestamp": "2022-02-25T19:16:32.462+00:00", "status": 404, "error": "Not Found", "path": "/servers/exchangeDL01/open-metadata/integration-daemon//users/peterprofile/integration-services/files-integrator/connectors/OakDeneLandingAreaFilesMonitor/configuration-properties" }

Response: { "timestamp": "2022-02-25T19:16:32.515+00:00", "status": 404, "error": "Not Found", "path": "/servers/exchangeDL01/open-metadata/integration-daemon//users/peterprofile/integration-services/files-integrator/connectors/configuration-properties" }

Response: { "timestamp": "2022-02-25T19:16:32.540+00:00", "status": 404, "error": "Not Found", "path": "/servers/exchangeDL01/open-metadata/integration-daemon//users/peterprofile/integration-services/files-integrator/connectors/OldMarketLandingAreaFilesMonitor/configuration-properties" }

* fails then in 2 - entity not found:

The guid for the provision-weekly-measurements-governance-action-service governance service is: 2e04aeb7-52e9-4233-b9b7-60b55441bbc3 OMAG-REPOSITORY-HANDLER-404-007 The GovernanceService entity with unique identifier 2e04aeb7-52e9-4233-b9b7-60b55441bbc3 is not found for method registerGovernanceServiceWithEngine of access service Governance Engine OMAS in open metadata server cocoMDS1, error message was: OMRS-REPOSITORY-404-002 The entity identified with guid 2e04aeb7-52e9-4233-b9b7-60b55441bbc3 passed on the getEntityDetail call is not known to the open metadata repository cocoMDS1

The system is unable to update information associated with the entity because none of the connected open metadata repositories recognize the entity's unique identifier.
The unique identifier of the entity is supplied by the caller. Verify that the caller's logic is correct, and that there are no errors being reported by the open metadata repository. Once all errors have been resolved, retry the request. Service registered as: move-file
```
There are other failures further down that have been previously noted, ie listing dir
```

As above, this was running in chart -- and as egeria evolved, it would be useful to fix up this lab to work on k8s.

(just rechecking locally)

planetf1 commented 2 years ago

These same errors are occuring locally.

@mandy-chessell I realise this lab is incomplete & not ready for use,though previously only the os.listdir was failing on k8s. We now have more failures. I presume caused by some refactoring. Can you confirm there's nothing of undue concern here?

planetf1 commented 2 years ago

@mandy-chessell Confirming this same failure occurs in 3.7 - though the lab is incomplete so assume work-in-progress.

mandy-chessell commented 2 years ago

No change has happened to it so not surprised it is still failing

planetf1 commented 1 year ago

In ad13e865f857eec36ca5466d1d4b079fc8aa63e3 I noticed the lab was updated and the in-dev label removed.

On testing, I get a failure at step 11:


waitForRunningGovernanceEngine(governDL01Name, governDL01PlatformName, governDL01PlatformURL, petersUserId, assetGovernanceEngineName)

printGovernanceEngineStatuses(governDL01Name, governDL01PlatformName, governDL01PlatformURL, petersUserId)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In [11], line 1
----> 1 waitForRunningGovernanceEngine(governDL01Name, governDL01PlatformName, governDL01PlatformURL, petersUserId, assetGovernanceEngineName)
      3 printGovernanceEngineStatuses(governDL01Name, governDL01PlatformName, governDL01PlatformURL, petersUserId)

NameError: name 'assetGovernanceEngineName' is not defined

This value is referred to but never set

mandy-chessell commented 1 year ago

You are right - I am still working on it - hopefully get it finished this week ... I was not expecting anyone to run the labs until the release.

I am changing it to work from the archives rather than defining the governance engine. I have harvested the pieces that manually harvested the engines and put them in the new development lab called governance-servers-lab. It would be interesting to see if it works in k8 - I have it running localy on my machine

planetf1 commented 1 year ago

I tried the updated lab (governance-server-operation) which failed at step.22

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [22], in <cell line: 21>()
     17     waitForRunningGovernanceAction(simpleMetadataCatalog, devPlatformName, devPlatformURL, erinsUserId, governanceActionGUID)
     18     printGovernanceAction(simpleMetadataCatalog, devPlatformName, devPlatformURL, erinsUserId, governanceActionGUID)
---> 21 addFileToLandingArea(filesSourceFolder, landingAreaDirectory, "1", "FTP Oak Dene Week 1")

Input In [22], in addFileToLandingArea(sourceFolder, destinationFolder, weekNumber, qualifiedName)
      7 governanceActionGUID = None
      8 governanceActionGUID = initiateGovernanceAction(simpleMetadataCatalog,
      9                                                 devPlatformName,
     10                                                 devPlatformURL,
   (...)
     15                                                 requestParameters,
     16                                                 "Governance Services Lab")
---> 17 waitForRunningGovernanceAction(simpleMetadataCatalog, devPlatformName, devPlatformURL, erinsUserId, governanceActionGUID)
     18 printGovernanceAction(simpleMetadataCatalog, devPlatformName, devPlatformURL, erinsUserId, governanceActionGUID)

File /var/folders/y0/dtxnsmfs0sz8_01kd0hhfjrc0000gn/T/ipykernel_81021/1004445229.py:131, in waitForRunningGovernanceAction(serverName, serverPlatformName, serverPlatformURL, userId, governanceActionGUID)
    129 while not checkGovernanceActionCompletion(governanceActionElement):
    130     governanceActionStatus = governanceActionElement.get('actionStatus')
--> 131     print(" ... Waiting for completion status, current status is: " + governanceActionStatus)
    132     time.sleep(1)
    133     governanceActionElement = getGovernanceAction(serverName, serverPlatformName, serverPlatformURL, userId, governanceActionGUID)

TypeError: can only concatenate str (not "NoneType") to str

Is this lab still in development? If so may be worth adding a statement to clarify for the user.

mandy-chessell commented 1 year ago

This is a different lab to the automated curation lab - should be a different issue. I am happy to fix - it should be working.

mandy-chessell commented 1 year ago

I had a think about it and I expect this test was run against the release 3.14 and not against main? If this is the case then it will fail. It uses many of the same functions as the automated curation lab that only work in 3.15-SNAPSHOT and later.

It is difficult to tell what the environment is from these comments.

I suggest the governance-server-operation lab has the same comment added to the top to say it is V3.15 or later.

planetf1 commented 1 year ago

Yes, I tested on 3.14 (as part of release) - I should have clarified. I will add that disclaimer

planetf1 commented 1 year ago

Confirmed - tested on 3.15-SNAPSHOT via k8s, fails as expected

This is because the file is written on one platform ,and we try to read on another

For this scenario we would probably expect coco to have a shared file server, and so that is probably what we need in the chart environment

planetf1 commented 1 year ago

Opened up odpi/egeria-charts#226 to add support for a shared filesystem for coco

mandy-chessell commented 1 year ago

In this notebook all access to the files is done by the same platform. The failure is in an call made by the jupyter notebook. The juypyter notebook needs to have access to the same file system as the platform(s).

planetf1 commented 1 year ago

@mandy-chessell sure. I was thinking generally about access from multiple containers :-) - the references issue should fallow the potential for sharing between any of the containers - clients (we don't have yet), egeria platform, jupyter

Another option is to allow for simple remote file access (ie via ssh). that's almost cleaner technically for a demo, but I think coco would find a shared fileserver useful for other scenarios....?

odpi / egeria-coco-labs

automated curation lab fails (containers, 2.8) #32