search

apache / incubator-heron

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
https://heron.apache.org/
Apache License 2.0
3.65k stars 599 forks source link

[3846] Refactoring of the K8s Shim #3847

Closed surahman closed 2 years ago

surahman commented 2 years ago

Refactoring of the K8s Shim

This issue is not related to a release and will be a code quality improvement. Closes #3846.

The V1Controller is a Shim to interface with the K8s cluster(s). It should only contain methods that support this functionality. All methods that communicate with the Cluster are retained in the KubernetesShim class. All methods that support the creation and configuration of the Stateful Sets are moved to the StatefulSet factory class. Data with cluster configurations and config maps are passed to the factory via the Configs container class.

Kubernetes Shim:


StatefulSet (Factory):



Deployment Testing:

Deployment :heavy_check_mark: Teardown :heavy_check_mark: Cluster in good state pre and post-deployment :heavy_check_mark:

Command ```bash ~/bin/heron submit kubernetes ~/.heron/examples/heron-api-examples.jar \ org.apache.heron.examples.api.AckingTopology acking \ --verbose \ --deploy-deactivated \ --config-property heron.kubernetes.executor.pod.template=pod-templ-executor.pod-template-executor.yaml \ --config-property heron.kubernetes.manager.pod.template=pod-templ-manager.pod-template-manager.yaml \ --config-property heron.kubernetes.manager.limits.cpu=2 \ --config-property heron.kubernetes.manager.limits.memory=3 \ --config-property heron.kubernetes.manager.requests.cpu=1 \ --config-property heron.kubernetes.manager.requests.memory=2 \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.claimName=OnDemand \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.storageClassName=storage-class-name-manager \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.accessModes=ReadWriteOnce,ReadOnlyMany \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.sizeLimit=256Gi \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.volumeMode=Block \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.path=path/to/mount/dynamic/volume \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-dynamic-volume.subPath=sub/path/to/mount/dynamic/volume \ \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.claimName=OnDemand \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.accessModes=ReadWriteOnce,ReadOnlyMany \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.sizeLimit=512Gi \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.volumeMode=Block \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.path=path/to/mount/static/volume \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-static-volume.subPath=sub/path/to/mount/static/volume \ \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-shared-volume.claimName=requested-claim-by-user \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-shared-volume.path=path/to/mount/shared/volume \ --config-property heron.kubernetes.manager.volumes.persistentVolumeClaim.manager-shared-volume.subPath=sub/path/to/mount/shared/volume \ \ --config-property heron.kubernetes.manager.volumes.emptyDir.manager-empty-dir.medium="Memory" \ --config-property heron.kubernetes.manager.volumes.emptyDir.manager-empty-dir.sizeLimit="50Mi" \ --config-property heron.kubernetes.manager.volumes.emptyDir.manager-empty-dir.path="empty/dir/path" \ --config-property heron.kubernetes.manager.volumes.emptyDir.manager-empty-dir.subPath="empty/dir/sub/path" \ --config-property heron.kubernetes.manager.volumes.emptyDir.manager-empty-dir.readOnly="true" \ \ --config-property heron.kubernetes.manager.volumes.hostPath.manager-host-path.type="File" \ --config-property heron.kubernetes.manager.volumes.hostPath.manager-host-path.pathOnHost="/dev/null" \ --config-property heron.kubernetes.manager.volumes.hostPath.manager-host-path.path="host/path/path" \ --config-property heron.kubernetes.manager.volumes.hostPath.manager-host-path.subPath="host/path/sub/path" \ --config-property heron.kubernetes.manager.volumes.hostPath.manager-host-path.readOnly="true" \ \ --config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.server="nfs-server.address" \ --config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.readOnly="true" \ --config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.pathOnNFS="/dev/null" \ --config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.path="nfs/path" \ --config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.subPath="nfs/sub/path" \ --config-property heron.kubernetes.manager.volumes.nfs.manager-nfs.readOnly="true" \ \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.claimName=OnDemand \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.storageClassName=storage-class-name-executor \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.accessModes=ReadWriteOnce,ReadOnlyMany \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.sizeLimit=256Gi \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.volumeMode=Block \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.path=path/to/mount/dynamic/volume \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-dynamic-volume.subPath=sub/path/to/mount/dynamic/volume \ \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.claimName=OnDemand \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.accessModes=ReadWriteOnce,ReadOnlyMany \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.sizeLimit=512Gi \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.volumeMode=Block \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.path=path/to/mount/static/volume \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-static-volume.subPath=sub/path/to/mount/static/volume \ \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-shared-volume.claimName=requested-claim-by-user \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-shared-volume.path=path/to/mount/shared/volume \ --config-property heron.kubernetes.executor.volumes.persistentVolumeClaim.executor-shared-volume.subPath=sub/path/to/mount/shared/volume ```
Stateful Sets __*Executor:*__ ```yaml apiVersion: v1 items: - apiVersion: apps/v1 kind: StatefulSet metadata: creationTimestamp: "2022-07-21T17:53:10Z" generation: 1 labels: app: heron topology: acking name: acking-executor namespace: default resourceVersion: "859" uid: a946bf3d-88df-4320-9752-40e6d0327959 spec: podManagementPolicy: Parallel replicas: 2 revisionHistoryLimit: 10 selector: matchLabels: app: heron topology: acking serviceName: acking template: metadata: annotations: prometheus.io/port: "8080" prometheus.io/scrape: "true" creationTimestamp: null labels: app: heron topology: acking spec: containers: - command: - sh - -c - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0.tar.gz . && SHARD_ID=$((${POD_NAME##*-} + 1)) && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor --topology-name=acking --topology-id=acking3255f000-6d02-476a-840d-9a3ca4338885 --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/* --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*: --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824 --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/* --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003 --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007 --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009' env: - name: HOST valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: var_one value: variable one - name: var_three value: variable three - name: var_two value: variable two image: apache/heron:testbuild imagePullPolicy: IfNotPresent name: executor ports: - containerPort: 5555 name: tcp-port-kept protocol: TCP - containerPort: 5556 name: udp-port-kept protocol: UDP - containerPort: 6001 name: server protocol: TCP - containerPort: 6002 name: tmanager-ctl protocol: TCP - containerPort: 6003 name: tmanager-stats protocol: TCP - containerPort: 6004 name: shell-port protocol: TCP - containerPort: 6005 name: metrics-mgr protocol: TCP - containerPort: 6006 name: scheduler protocol: TCP - containerPort: 6007 name: metrics-cache-m protocol: TCP - containerPort: 6008 name: metrics-cache-s protocol: TCP - containerPort: 6009 name: ckptmgr protocol: TCP resources: limits: cpu: "3" memory: 4Gi requests: cpu: "3" memory: 4Gi securityContext: allowPrivilegeEscalation: false terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: path/to/mount/dynamic/volume name: executor-dynamic-volume subPath: sub/path/to/mount/dynamic/volume - mountPath: path/to/mount/shared/volume name: executor-shared-volume subPath: sub/path/to/mount/shared/volume - mountPath: path/to/mount/static/volume name: executor-static-volume subPath: sub/path/to/mount/static/volume - mountPath: /shared_volume name: shared-volume - image: alpine imagePullPolicy: Always name: sidecar-container resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /shared_volume name: shared-volume dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 0 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 10 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 10 volumes: - name: executor-shared-volume persistentVolumeClaim: claimName: requested-claim-by-user - emptyDir: {} name: shared-volume updateStrategy: rollingUpdate: partition: 0 type: RollingUpdate volumeClaimTemplates: - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: null labels: onDemand: "true" topology: acking name: executor-dynamic-volume spec: accessModes: - ReadWriteOnce - ReadOnlyMany resources: requests: storage: 256Gi storageClassName: storage-class-name-executor volumeMode: Block status: phase: Pending - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: null labels: onDemand: "true" topology: acking name: executor-static-volume spec: accessModes: - ReadWriteOnce - ReadOnlyMany resources: requests: storage: 512Gi storageClassName: "" volumeMode: Block status: phase: Pending status: collisionCount: 0 currentReplicas: 2 currentRevision: acking-executor-69684dd444 observedGeneration: 1 replicas: 2 updateRevision: acking-executor-69684dd444 updatedReplicas: 2 ``` __*Manager:*__ ```yaml - apiVersion: apps/v1 kind: StatefulSet metadata: creationTimestamp: "2022-07-21T17:53:10Z" generation: 1 labels: app: heron topology: acking name: acking-manager namespace: default resourceVersion: "867" uid: 83590047-23dc-44d4-9244-23176e62f8bd spec: podManagementPolicy: Parallel replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: heron topology: acking serviceName: acking template: metadata: annotations: prometheus.io/port: "8080" prometheus.io/scrape: "true" creationTimestamp: null labels: app: heron topology: acking spec: containers: - command: - sh - -c - './heron-core/bin/heron-downloader-config kubernetes && ./heron-core/bin/heron-downloader distributedlog://zookeeper:2181/heronbkdl/acking-saad-tag-0.tar.gz . && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID} && ./heron-core/bin/heron-executor --topology-name=acking --topology-id=acking3255f000-6d02-476a-840d-9a3ca4338885 --topology-defn-file=acking.defn --state-manager-connection=zookeeper:2181 --state-manager-root=/heron --state-manager-config-file=./heron-conf/statemgr.yaml --tmanager-binary=./heron-core/bin/heron-tmanager --stmgr-binary=./heron-core/bin/heron-stmgr --metrics-manager-classpath=./heron-core/lib/metricsmgr/* --instance-jvm-opts="LVhYOitIZWFwRHVtcE9uT3V0T2ZNZW1vcnlFcnJvcg(61)(61)" --classpath=heron-api-examples.jar --heron-internals-config-file=./heron-conf/heron_internals.yaml --override-config-file=./heron-conf/override.yaml --component-ram-map=exclaim1:1073741824,word:1073741824 --component-jvm-opts="" --pkg-type=jar --topology-binary-file=heron-api-examples.jar --heron-java-home=$JAVA_HOME --heron-shell-binary=./heron-core/bin/heron-shell --cluster=kubernetes --role=saad --environment=default --instance-classpath=./heron-core/lib/instance/* --metrics-sinks-config-file=./heron-conf/metrics_sinks.yaml --scheduler-classpath=./heron-core/lib/scheduler/*:./heron-core/lib/packing/*:./heron-core/lib/statemgr/* --python-instance-binary=./heron-core/bin/heron-python-instance --cpp-instance-binary=./heron-core/bin/heron-cpp-instance --metricscache-manager-classpath=./heron-core/lib/metricscachemgr/* --metricscache-manager-mode=disabled --is-stateful=false --checkpoint-manager-classpath=./heron-core/lib/ckptmgr/*:./heron-core/lib/statefulstorage/*: --stateful-config-file=./heron-conf/stateful.yaml --checkpoint-manager-ram=1073741824 --health-manager-mode=disabled --health-manager-classpath=./heron-core/lib/healthmgr/* --shard=$SHARD_ID --server-port=6001 --tmanager-controller-port=6002 --tmanager-stats-port=6003 --shell-port=6004 --metrics-manager-port=6005 --scheduler-port=6006 --metricscache-manager-server-port=6007 --metricscache-manager-stats-port=6008 --checkpoint-manager-port=6009' env: - name: HOST valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: var_one_manager value: variable one on manager - name: var_three_manager value: variable three on manager - name: var_two_manager value: variable two on manager image: apache/heron:testbuild imagePullPolicy: IfNotPresent name: manager ports: - containerPort: 6001 name: server protocol: TCP - containerPort: 6002 name: tmanager-ctl protocol: TCP - containerPort: 6003 name: tmanager-stats protocol: TCP - containerPort: 6004 name: shell-port protocol: TCP - containerPort: 6005 name: metrics-mgr protocol: TCP - containerPort: 6006 name: scheduler protocol: TCP - containerPort: 6007 name: metrics-cache-m protocol: TCP - containerPort: 6008 name: metrics-cache-s protocol: TCP - containerPort: 6009 name: ckptmgr protocol: TCP - containerPort: 7775 name: tcp-port-kept protocol: TCP - containerPort: 7776 name: udp-port-kept protocol: UDP resources: limits: cpu: "2" memory: "3" requests: cpu: "1" memory: "2" securityContext: allowPrivilegeEscalation: false terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: path/to/mount/dynamic/volume name: manager-dynamic-volume subPath: sub/path/to/mount/dynamic/volume - mountPath: empty/dir/path name: manager-empty-dir readOnly: true subPath: empty/dir/sub/path - mountPath: host/path/path name: manager-host-path readOnly: true subPath: host/path/sub/path - mountPath: nfs/path name: manager-nfs readOnly: true subPath: nfs/sub/path - mountPath: path/to/mount/shared/volume name: manager-shared-volume subPath: sub/path/to/mount/shared/volume - mountPath: path/to/mount/static/volume name: manager-static-volume subPath: sub/path/to/mount/static/volume - mountPath: /shared_volume/manager name: shared-volume-manager - image: alpine imagePullPolicy: Always name: manager-sidecar-container resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /shared_volume/manager name: shared-volume-manager dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 0 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 10 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 10 volumes: - emptyDir: medium: Memory sizeLimit: 50Mi name: manager-empty-dir - hostPath: path: /dev/null type: File name: manager-host-path - name: manager-nfs nfs: path: /dev/null readOnly: true server: nfs-server.address - name: manager-shared-volume persistentVolumeClaim: claimName: requested-claim-by-user - emptyDir: {} name: shared-volume-manager updateStrategy: rollingUpdate: partition: 0 type: RollingUpdate volumeClaimTemplates: - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: null labels: onDemand: "true" topology: acking name: manager-static-volume spec: accessModes: - ReadWriteOnce - ReadOnlyMany resources: requests: storage: 512Gi storageClassName: "" volumeMode: Block status: phase: Pending - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: null labels: onDemand: "true" topology: acking name: manager-dynamic-volume spec: accessModes: - ReadWriteOnce - ReadOnlyMany resources: requests: storage: 256Gi storageClassName: storage-class-name-manager volumeMode: Block status: phase: Pending status: collisionCount: 0 currentReplicas: 1 currentRevision: acking-manager-86689946f9 observedGeneration: 1 replicas: 1 updateRevision: acking-manager-86689946f9 updatedReplicas: 1 ```
Persistent Volume Claims ```yaml apiVersion: v1 items: - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: "2022-07-21T17:53:10Z" finalizers: - kubernetes.io/pvc-protection labels: app: heron onDemand: "true" topology: acking name: executor-dynamic-volume-acking-executor-0 namespace: default resourceVersion: "838" uid: 2fe7e176-5821-46fb-bc09-fd2e76793690 spec: accessModes: - ReadWriteOnce - ReadOnlyMany resources: requests: storage: 256Gi storageClassName: storage-class-name-executor volumeMode: Block status: phase: Pending - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: "2022-07-21T17:53:10Z" finalizers: - kubernetes.io/pvc-protection labels: app: heron onDemand: "true" topology: acking name: executor-dynamic-volume-acking-executor-1 namespace: default resourceVersion: "845" uid: cb8b7907-7fba-4603-9864-3da8f9e7c19c spec: accessModes: - ReadWriteOnce - ReadOnlyMany resources: requests: storage: 256Gi storageClassName: storage-class-name-executor volumeMode: Block status: phase: Pending - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: "2022-07-21T17:53:10Z" finalizers: - kubernetes.io/pvc-protection labels: app: heron onDemand: "true" topology: acking name: executor-static-volume-acking-executor-0 namespace: default resourceVersion: "839" uid: 8dd24cec-bd3c-4a00-b5e3-ec6087535b81 spec: accessModes: - ReadWriteOnce - ReadOnlyMany resources: requests: storage: 512Gi storageClassName: "" volumeMode: Block status: phase: Pending - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: "2022-07-21T17:53:10Z" finalizers: - kubernetes.io/pvc-protection labels: app: heron onDemand: "true" topology: acking name: executor-static-volume-acking-executor-1 namespace: default resourceVersion: "849" uid: 9e67b08f-df1c-454c-8eda-fef5a46d7fd5 spec: accessModes: - ReadWriteOnce - ReadOnlyMany resources: requests: storage: 512Gi storageClassName: "" volumeMode: Block status: phase: Pending - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: "2022-07-21T17:53:10Z" finalizers: - kubernetes.io/pvc-protection labels: app: heron onDemand: "true" topology: acking name: manager-dynamic-volume-acking-manager-0 namespace: default resourceVersion: "856" uid: 2457796c-2274-4e3a-b786-324895b8e979 spec: accessModes: - ReadWriteOnce - ReadOnlyMany resources: requests: storage: 256Gi storageClassName: storage-class-name-manager volumeMode: Block status: phase: Pending - apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: "2022-07-21T17:53:10Z" finalizers: - kubernetes.io/pvc-protection labels: app: heron onDemand: "true" topology: acking name: manager-static-volume-acking-manager-0 namespace: default resourceVersion: "850" uid: 7174e5b4-1079-48ee-90fb-0ddc8f08d6d0 spec: accessModes: - ReadWriteOnce - ReadOnlyMany resources: requests: storage: 512Gi storageClassName: "" volumeMode: Block status: phase: Pending kind: List metadata: resourceVersion: "" selfLink: "" ```
surahman commented 2 years ago

Thank you @joshfischer1108 and @nicknezis