search

flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.76k stars 656 forks source link

[FlyteCTL Bug] failed to perform fast register on sandbox #2017

Open wenshutang opened 2 years ago

wenshutang commented 2 years ago

Describe the feature/command for FlyteCTL

I'm having issues when attempting to fast register

Packinging using pyflyte

pyflyte -c sandbox.config --pkgs basic_wfs serialize --local-source-root ./ --image basic_wfs:v0 fast workflows -f _pb_output/

Fast register

flytectl register files  _pb_output/* -d development  -p piccadilly --version fastv2.0 Error: please check your Storage Config. It failed while uploading the source code. Failed to write data [56b] to path [fast/fastv2.0-fast42058067df98d17a434dc57fa62e2f3d.tar.gz].: PutObject, putting object: SerializationError: failed to decode REST XML response
        status code: 200, request id: 
caused by: XML syntax error on line 1: element <link> closed by </head>

I set up my flyte sandbox cluster using all default settings.

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///localhost:30081
  authType: Pkce
  insecure: true
logger:
  show-source: true
  level: 0
storage:
  connection:
    access-key: minio
    auth-type: accesskey
    disable-ssl: true
    endpoint: http://localhost:30084
    region: us-east-1
    secret-key: miniostorage
  type: minio
  container: "my-s3-bucket"

What am I missing here?

Provide a possible output or UX example

To properly build and fast register workflows to tighten iteration loops

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

welcome[bot] commented 2 years ago

Thank you for opening your first issue here! 🛠

kumare3 commented 2 years ago

@wenshutang thank you for the bug, hmm your config at first glance looks fine. Cc @pmahindrakar-oss / @evalsocket

pmahindrakar-oss commented 2 years ago

@wenshutang can you help understand the platform where you are installing flytectl There is reportedly an issue on Centos https://github.com/chrislusf/seaweedfs/pull/1402 with List api in go client . Would be helpful to know more before we dig deeper

wenshutang commented 2 years ago

@pmahindrakar-oss thanks for getting back to me. I've deployed sandbox on docker desktop for mac, using the docker k8s cluster.

Running flytectl on osx Big Sur

flytesnacks git:(master) ✗ flytectl version 

 A new release of flytectl is available: 0.3.22 → v0.4.9 
To upgrade, run: brew update && brew upgrade flytectl 
https://github.com/flyteorg/flytectl/releases/tag/v0.4.9 

{
  "App": "flytectl",
  "Build": "68db2d5",
  "Version": "0.3.22",
  "BuildTime": "2022-01-05 14:25:28.362427 -0800 PST m=+0.025125864"
}{
  "App": "controlPlane",
  "Build": "a93b91e",
  "Version": "v0.6.64",
  "BuildTime": "2021-12-28 06:10:08.198858865 +0000 UTC m=+0.442487979"
}
wenshutang commented 2 years ago

Some more information, after upgrading to flytectl v0.4.9. I'm still getting the same error. I am able to access the minio console and have successfully registered many workflows.

pmahindrakar-oss commented 2 years ago

@wenshutang I performed fast registration operation on Bigsur aswell and couldn't reproduce this issue. Used the examples from cookbook/core.

Can you help run the same examples and let me know what the outcome. https://github.com/flyteorg/flytesnacks/tree/master/cookbook/core Using the wrapper make scripts to do the same

register the files using flytectl

register the fast version using flytectl

pmahindrakar-oss commented 2 years ago

The error could be due to numerous reasons https://github.com/aws/aws-sdk-go/blob/4fd4b72d1a40237285232f1b16c1d13de4f1220d/private/protocol/restxml/restxml.go#L57

Can you print out the minio server version being used.

The sandbox enviornment i deployed is using the following which seems to be working with client used in flytectl

minio-65fbb69485-xtrqz:/opt/bitnami/minio/bin$ minio --version
minio version DEVELOPMENT.2021-10-13T00-23-17Z
wenshutang commented 2 years ago

The two make steps completed successfully when packaging cookbook/core. When trying to fast register, I am hitting the same error.

flytectl register files  _pb_output/* -d development  -p flytesnacks --version fastv1.0
Error: please check your Storage Config. It failed while uploading the source code. Failed to write data [56b] to path [fast/fastv1.0-fastcb901fa11e1b7ed864255bf234b21b93.tar.gz].: PutObject, putting object: SerializationError: failed to decode REST XML response
        status code: 200, request id: 
caused by: XML syntax error on line 1: element <link> closed by </head>

Looks like I am on the same version of minio.

$ minio --version
minio version DEVELOPMENT.2021-10-13T00-23-17Z
pmahindrakar-oss commented 2 years ago

This is strange. This is failing while uploading the fast registered source file archive for you.

Are you able to perform this upload manually to minio

eg : 

(base) ➜  temp export AWS_ACCESS_KEY_ID="minio"
(base) ➜  temp export AWS_SECRET_ACCESS_KEY="miniostorage"
(base) ➜  temp aws --endpoint-url http://localhost:30084 s3 cp fast2e28c9f788423a13ca63c0ed7e3e7f6e.tar.gz s3://my-s3-bucket/temp/fast2e28c9f788423a13ca63c0ed7e3e7f6e.tar.gz
upload: ./fast2e28c9f788423a13ca63c0ed7e3e7f6e.tar.gz to s3://my-s3-bucket/temp/fast2e28c9f788423a13ca63c0ed7e3e7f6e.tar.gz

Also can you dump flytekit version used during make fast_serialize

pip list  |grep flytekit
wenshutang commented 2 years ago

@pmahindrakar-oss Appreciate the follow up. aws cp is indeed failing.

My flytekit version

pip list | grep flytekit
flytekit               0.25.0b3
aws --endpoint-url http://localhost:30084 s3 cp _pb_output/fastcb901fa11e1b7ed864255bf234b21b93.tar.gz s3://my-s3-bucket/temp/fastcb901fa11e1b7ed864255bf234b21b93.tar.gz
upload failed: _pb_output/fastcb901fa11e1b7ed864255bf234b21b93.tar.gz to s3://my-s3-bucket/temp/fastcb901fa11e1b7ed864255bf234b21b93.tar.gz Unable to parse response (syntax error: line 1, column 0), invalid XML received. Further retries may succeed:
b'<!doctype html><html lang="en"><head><meta charset="utf-8"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#081C42" media="(prefers-color-scheme: light)"/><meta name="theme-color" content="#081C42" media="(prefers-color-scheme: dark)"/><meta name="description" content="MinIO Console"/><link href="/styles/root-styles.css" rel="stylesheet"/><link rel="apple-touch-icon" sizes="180x180" href="/apple-icon-180x180.png"/><link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png"/><link rel="icon" type="image/png" sizes="96x96" href="/favicon-96x96.png"/><link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png"/><link rel="manifest" href="/manifest.json"/><link rel="mask-icon" href="/safari-pinned-tab.svg" color="#3a4e54"/><title>MinIO Console</title><link href="/static/css/2.5b1f144e.chunk.css" rel="stylesheet"><link href="/static/css/main.e33a67ba.chunk.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"><div id="loader-block"><svg class="loader-svg-container" viewBox="22 22 44 44"><circle class="loader-style MuiCircularProgress-circle MuiCircularProgress-circleIndeterminate" cx="44" cy="44" r="20.2" fill="none" stroke-width="3.6"></circle></svg></div></div><script>!function(e){function r(r){for(var n,l,i=r[0],a=r[1],p=r[2],c=0,s=[];c<i.length;c++)l=i[c],Object.prototype.hasOwnProperty.call(o,l)&&o[l]&&s.push(o[l][0]),o[l]=0;for(n in a)Object.prototype.hasOwnProperty.call(a,n)&&(e[n]=a[n]);for(f&&f(r);s.length;)s.shift()();return u.push.apply(u,p||[]),t()}function t(){for(var e,r=0;r<u.length;r++){for(var t=u[r],n=!0,i=1;i<t.length;i++){var a=t[i];0!==o[a]&&(n=!1)}n&&(u.splice(r--,1),e=l(l.s=t[0]))}return e}var n={},o={1:0},u=[];function l(r){if(n[r])return n[r].exports;var t=n[r]={i:r,l:!1,exports:{}};return e[r].call(t.exports,t,t.exports,l),t.l=!0,t.exports}l.m=e,l.c=n,l.d=function(e,r,t){l.o(e,r)||Object.defineProperty(e,r,{enumerable:!0,get:t})},l.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},l.t=function(e,r){if(1&r&&(e=l(e)),8&r)return e;if(4&r&&"object"==typeof e&&e&&e.__esModule)return e;var t=Object.create(null);if(l.r(t),Object.defineProperty(t,"default",{enumerable:!0,value:e}),2&r&&"string"!=typeof e)for(var n in e)l.d(t,n,function(r){return e[r]}.bind(null,n));return t},l.n=function(e){var r=e&&e.__esModule?function(){return e.default}:function(){return e};return l.d(r,"a",r),r},l.o=function(e,r){return Object.prototype.hasOwnProperty.call(e,r)},l.p="./";var i=this["webpackJsonpportal-ui"]=this["webpackJsonpportal-ui"]||[],a=i.push.bind(i);i.push=r,i=i.slice();for(var p=0;p<i.length;p++)r(i[p]);var f=a;t()}([])</script><script src="/static/js/2.dd760fd2.chunk.js"></script><script src="/static/js/main.e81f26fe.chunk.js"></script></body></html>'
pmahindrakar-oss commented 2 years ago

@wenshutang the minio response mentions invalid XML received. Further retries may succeed: Did you try calling the same command multiple times to see if it helps.

Also is it possible for you to run the same command on an empty file and call s3 cp to upload this to minio from the commandline.

this looks moreover a minio issue. The other component versions you mentioned have been working for me. One another thing you could try is by updating to latest minio server version https://github.com/minio/minio/tags

And update it here https://github.com/flyteorg/flyte/blob/35e261697ea640355d8aa23926b0fec2db13eb0f/charts/flyte/values.yaml#L706

You can then run helm install steps to update to the latest charts with updated minio from within flyte/charts/flyte directory

helm install -n flyte -f values.yaml --create-namespace flyte .
kumare3 commented 2 years ago

@wenshutang did you get it working, or is it still failing for you?

github-actions[bot] commented 1 year ago

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

github-actions[bot] commented 1 year ago

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

github-actions[bot] commented 3 months ago

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. Thank you for your contribution and understanding! 🙏