Creating datalinks in Enso Cloud programmatically

[radeusgd]

Currently there's no way to create a datalink from Enso code.

It is not only useful for users and library creators but it is also needed so that our datalink tests could be made self-sufficient, not expecting an existing setup on the test account.

[radeusgd]

I've already adapted our data link tests to create the datalinks themelves... but I forgot it's not yet possible:

Index: test/Base_Tests/data/datalinks/example-enso-file.datalink
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/test/Base_Tests/data/datalinks/example-enso-file.datalink b/test/Base_Tests/data/datalinks/example-enso-file.datalink
--- a/test/Base_Tests/data/datalinks/example-enso-file.datalink (revision 113524b039474a92844622232e9572b30f26f248)
+++ b/test/Base_Tests/data/datalinks/example-enso-file.datalink (date 1712768831228)
@@ -1,5 +1,5 @@
 {
     "type": "Enso_File",
     "libraryName": "Standard.Base",
-    "path": "enso://PLACEHOLDER_ORG_NAME/test-directory/another.txt"
+    "path": "enso://PLACEHOLDER_ORG_NAME/PLACEHOLDER_PATH"
 }
Index: test/Base_Tests/src/Network/Enso_Cloud/Cloud_Data_Link_Spec.enso
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/test/Base_Tests/src/Network/Enso_Cloud/Cloud_Data_Link_Spec.enso b/test/Base_Tests/src/Network/Enso_Cloud/Cloud_Data_Link_Spec.enso
--- a/test/Base_Tests/src/Network/Enso_Cloud/Cloud_Data_Link_Spec.enso  (revision 113524b039474a92844622232e9572b30f26f248)
+++ b/test/Base_Tests/src/Network/Enso_Cloud/Cloud_Data_Link_Spec.enso  (date 1712769039360)
@@ -11,24 +11,40 @@

 import project.Network.Enso_Cloud.Cloud_Tests_Setup.Cloud_Tests_Setup

-add_specs suite_builder setup:Cloud_Tests_Setup = setup.with_prepared_environment <|
+type Temporary_Directory
+    Value ~get
+
+    make -> Temporary_Directory = Temporary_Directory.Value <|
+        name = "test-run-Data_Link-"+(Date_Time.now.format "yyyy-MM-dd_HHmmss.fV" . replace "/" "|")
+        root = Panic.rethrow <| Enso_File.root.create_directory name
+
+        Panic.rethrow <| Data_Link_Format.write_raw_config (root / "TestDataLink-HTTP.datalink") http_data_link_content
+        Panic.rethrow <| Data_Link_Format.write_raw_config (root / "TestDataLink-S3.datalink") s3_data_link_content
+        my_target_enso_file = "Hello Enso Cloud File!".write (root / "my-file.txt")
+        Panic.rethrow <| Data_Link_Format.write_raw_config (root / "TestDataLink-EnsoFile.datalink") (enso_file_data_link_content.replace "<PATH>" my_target_enso_file.path)
+
+        root
+
+    cleanup self = self.get.delete_if_exists
+
+add_specs suite_builder setup:Cloud_Tests_Setup =
     suite_builder.group "DataLinks in Enso Cloud" pending=setup.real_cloud_pending group_builder->
+        test_root = Temporary_Directory.make
+        group_builder.teardown test_root.cleanup
+
         group_builder.specify "should be able to access an example HTTP data-link" <|
-            ## We assume that the cloud has a sample data-link called `TestDataLink` present
-               TODO in future iterations this file will be created by the test suite itself, to make it self-contained
-            assets = Enso_File.root.list
-            datalink = assets.find a-> a.name.starts_with "TestDataLink-HTTP"
+            datalink = test_root.get / "TestDataLink-HTTP.datalink"
             datalink.should_be_a Enso_File
             datalink.asset_type.should_equal Enso_Asset_Type.Data_Link

-            # We assume that the datalink points to https://raw.githubusercontent.com/enso-org/enso/develop/test/Base_Tests/data/sample.txt
-            r = datalink.read
-            r.should_be_a Text
-            r.should_contain "Cupcake"
+            Test.with_retries <|
+                # We assume that the datalink points to
+                r = datalink.read
+                r.should_be_a Text
+                r.should_contain "Cupcake"

         group_builder.specify "will report which library is missing if a datalink relying on other library is accessed" <|
-            assets = Enso_File.root.list
-            datalink = assets.find a-> a.name.starts_with "TestDataLink-S3"
+            datalink = test_root.get / "TestDataLink-S3.datalink"
             datalink.should_be_a Enso_File
             datalink.asset_type.should_equal Enso_Asset_Type.Data_Link

@@ -44,14 +60,12 @@
                 Data_Link_Format.read_raw_config temp_file

         group_builder.specify "should be able to read a local datalink to an Enso File" <|
-            datalink = replace_org_name_in_data_link (enso_project.data / "datalinks" / "example-enso-file.datalink")
-            datalink.read . should_equal "Hello Another!"
+            datalink = replace_placeholder_path (enso_project.data / "datalinks" / "example-enso-file.datalink") (test_root.get / "my-file.txt")
+            datalink.read . should_equal "Hello Enso Cloud File!"

         group_builder.specify "should be able to read a datalink in the Cloud to Enso File" <|
-            # TODO currently this link is created manually, later we should be generating it as part of the test
-            datalink = Enso_File.new "enso://"+Enso_User.current.organization_name+"/TestDataLink-EnsoFile"
-            datalink.read . should_equal "Hello Another!"
-
+            datalink = Enso_File.new test_root.get.path+"/TestDataLink-EnsoFile.datalink"
+            datalink.read . should_equal "Hello Enso Cloud File!"

 main filter=Nothing =
     setup = Cloud_Tests_Setup.prepare
@@ -61,9 +75,34 @@

 ## Reads the datalink as plain text and replaces the placeholder organization name.
-replace_org_name_in_data_link base_file =
-    content = Data_Link_Format.read_raw_config base_file
-    org_name = Enso_User.current.organization_name
-    new_content = content.replace "PLACEHOLDER_ORG_NAME" org_name
-    temp_file = File.create_temporary_file prefix=base_file.name suffix=base_file.extension
+replace_placeholder_path datalink_file target_file =
+    content = Data_Link_Format.read_raw_config datalink_file
+    new_content = content.replace "enso://PLACEHOLDER_ORG_NAME/PLACEHOLDER_PATH" target_file.path
+    temp_file = File.create_temporary_file prefix=datalink_file.name suffix=datalink_file.extension
     Data_Link_Format.write_raw_config temp_file new_content replace_existing=True . if_not_error temp_file
+
+http_data_link_content = """
+    {
+        "type": "HTTP",
+        "libraryName": "Standard.Base",
+        "method": "GET",
+        "uri": "https://raw.githubusercontent.com/enso-org/enso/develop/test/Base_Tests/data/sample.txt"
+    }
+
+s3_data_link_content = """
+    {
+      "type": "S3",
+      "libraryName": "Standard.AWS",
+      "uri": "s3://enso-data-samples/examples/folder 2/hello.txt",
+      "auth": {
+        "type": "aws_auth",
+        "subType": "default"
+      }
+    }
+
+enso_file_data_link_content = """
+    {
+        "type": "Enso_File",
+        "libraryName": "Standard.Base",
+        "path": "<PATH>"
+    }

The goal of this ticket would be to apply this patch and ensure it actually works - we need to intercept a with_output_stream that targets a .datalink and ensure it instead performs a different POST request - to the create_connector endpoint instead of upload_file.

[enso-bot[bot]]

Radosław Waśko reports a new STANDUP for the provided date (2024-05-09):

Progress: Work on creating datalinks programmatically: updated the tests. Added the logic for creating datalink in Enso Cloud. Debugging to make it work. Work on types: learning details of method resolution, thinking how to replicate the logic in static passes. It should be finished by 2024-05-14.

Next Day: Next day I will be working on the same task. Fix the datalink creation logic. Add a method to save a datalink to Postgres connection.

[enso-bot[bot]]

Radosław Waśko reports a new STANDUP for yesterday (2024-05-13):

Progress: Debugging datalink creation. Got it fixed finally. Found a problem with wrong logic when parsing responses from cloud that went undetected because dataflow error was swallowed. Fixed the problem and ensured propagation in that case. Reported #9934. Got the tests passing with now no setup needed on cloud side - a step towards being able to run the Cloud tests on a CI. Implemented logic for saving Postgres connection as a datalink, promoting a plaintext password to a secret. It should be finished by 2024-05-14.

Next Day: Next day I will be working on the same task. Test the Postgres save-as-datalink logic. Put up a PR. Start next tasks.

[enso-bot[bot]]

Radosław Waśko reports a new STANDUP for yesterday (2024-05-14):

Progress: Debugging remaining issues in the PR. Fixed error propagation in JS_Object.from_pairs. Fixed reading raw datalink data from Cloud. Fixed datalink metadata getters. PR ready. It should be finished by 2024-05-14.

Next Day: Next day I will be working on the #9952 task. Start work on next tasks. Create PR for cloud path resolver update.