Open minhqdao opened 1 year ago
When @henilp105 & I created the upload API, it was supposed that information such as tags
related to the project, dependencies
will also come from frontend.
Should I remove these fields for now from the backend or just use a default value for all the packages. So, that the other APIs which are trying to fetch it does not break or can you send this data from fpm as well ?
The data that we might need but it's not necessary right now :
description, tags and dependencies
The data which is necessary and needs to be sent from fpm includes:
package_name, package_version, tarball, upload_token, license
cc: @minhqdao @henilp105 @perazz
Should we add time based expiry on the token or naming the tokens , as we would also have to add the functionality to revoke the tokens or expire them after certain time period.
Exactly @henilp105 , That's how it should be. But considering the time-frame we can maybe add later as well.
So, are we currently making the tokens irrevocable tokens which the user won't be able to re-view after creating them once. ( every time the user requests a new token would be generated. )
So, are we currently making the tokens irrevocable tokens which the user won't be able to re-view after creating them once. ( every time the user requests a new token would be generated. )
Yes, I think that works for now. But, we have to keep in mind that tokens should expire after certain amount of time & also an API + UI for deleting the generated tokens should be good.
The post request for the API should be sent in the form of form-data like this:
{
"package_name": "some_package_name",
"package_version": "X.X.X",
"package_license": "MIT",
"upload_token": "randomstringhere",
"tarball": <attach_tar.gz>
}
I have changed the deployment to use vercel for now because it is fast than render.com & not causing any errors now. Url - https://registry-apis.vercel.app/
Endpoint :
<base-url>/packages
request-type - POST
I will separately DM the upload token that @minhqdao & @perazz can use to make the requests.
Should we add time based expiry on the token or naming the tokens , as we would also have to add the functionality to revoke the tokens or expire them after certain time period.
We have a createdDate
of the token and therefore do not need to add an expiredDate
or an expirationTimeframe
. Instead, when the user wants to use the uploadToken
, we check, e.g. when the token is supposed to be valid for one week, if the token was created less than a week ago. With that you only need the createdDate
to check for validity.
The post request for the API should be sent in the form of form-data like this:
{ "package_name": "some_package_name", "package_version": "X.X.X", "package_license": "MIT", "upload_token": "randomstringhere", "tarball": <attach_tar.gz> }
I have changed the deployment to use vercel for now because it is fast than render.com & not causing any errors now. Url - https://registry-apis.vercel.app/
Endpoint :
<base-url>/packages request-type - POST
I will separately DM the upload token that @minhqdao & @perazz can use to make the requests.
I'm not quite sure what you mean by sending form data. I hope we are not mixing frontend with backend techniques here. I will send data via the -d
parameter (curl
) or via --post-data
(wget
). It is the default way of sending data through a POST
request.
We had this discussion already when we were thinking about sending a list of cached versions to the backend to determine whether we need to download a version or not.
When @henilp105 & I created the upload API, it was supposed that information such as
tags
related to the project,dependencies
will also come from frontend.Should I remove these fields for now from the backend or just use a default value for all the packages. So, that the other APIs which are trying to fetch it does not break or can you send this data from fpm as well ?
The data that we might need but it's not necessary right now :
description, tags and dependencies
The data which is necessary and needs to be sent from fpm includes:
package_name, package_version, tarball, upload_token, license
cc: @minhqdao @henilp105 @perazz
Yes, let's just ignore the optional fields for now and deliver them as soon as possible.
We have a
createdDate
of the token and therefore do not need to add anexpiredDate
or anexpirationTimeframe
. Instead, when the user wants to use theuploadToken
, we check, e.g. when the token is supposed to be valid for one week, if the token was created less than a week ago. With that you only need thecreatedDate
to check for validity.
I would agree here. This can be checked in the upload API itself.
I'm not quite sure what you mean by sending form data. I hope we are not mixing frontend with backend techniques here. I will send data via the
-d
parameter (curl
) or via--post-data
(wget
). It is the default way of sending data through aPOST
request.
There must be a way in curl
or wget
to specify the headers in which you mention content-type as "multipart/form-data" ?
I am thinking to use multipart/form-data because it will enable the upload of tarball files.
I'm not quite sure what you mean by sending form data. I hope we are not mixing frontend with backend techniques here. I will send data via the
-d
parameter (curl
) or via--post-data
(wget
). It is the default way of sending data through aPOST
request.There must be a way in
curl
orwget
to specify the headers in which you mention content-type as "multipart/form-data" ?I am thinking to use multipart/form-data because it will enable the upload of tarball files.
Of course we could change the content type in the header to multipart/form-data
. But in my understanding, that is actually a frontend technique for websites that you can use for convenience. The general way to send data to an api is using the data
(-d
) parameter, not the form
(https://www.outsystems.com/blog/posts/consuming-multipart-form-data-rest-method/#multipart-form-data). Since we want a general api that connects to both websites and a CLI tool such as fpm
, I think we should the data
option, not the form
.
Of course we could change the content type in the header to
multipart/form-data
. But in my understanding, that is actually a frontend technique for websites that you can use for convenience. The general way to send data to an api is using thedata
(-d
) parameter, not theform
(https://www.outsystems.com/blog/posts/consuming-multipart-form-data-rest-method/#multipart-form-data). Since we want a general api that connects to both websites and a CLI tool such asfpm
, I think we should thedata
option, not theform
.
We could use the get_json()
parameter on the backend as well and then you could send the data with -d parameter but that was the case when only key value pairs are there. In our case, we have a file (along with key value pairs), which basically is making us to apply some constraints. https://flask.palletsprojects.com/en/2.2.x/patterns/fileuploads/
We are trying to access the file i.e tarball using request.files
and for that to work frontend website/cli should attach the data as multipart-form.
I read the blog shared above, it also mentions "The HTTP content type for this format is “application/x-www-form-urlencoded.” This works fine for simple input fields, but web pages may also allow users to upload one or more files. To support this, another format was created in the early days of the web, and it is known by its content type, which is “multipart/form-data.”
I see, if multipart/form-data
allows the simultaneous upload of files and data, then we'll stick to it. Thanks for the clarification. 🙏🏽
Okay, But now I see you can use something like this to upload as well.
curl -d @path/to/data.json https://reqbin.com/echo/post/json
Looks like there is another way. I would suggest you can use multipart/form data for now. And I try sending a curl request to the API and see if I can change that. But, if it works then I think it will be a minimal change on backend and maybe on fpm as well.
Yes, I also think it's fine to stick to multiform/form-data
for now (or forever).
Yes, I also think it's fine to stick to
multiform/form-data
for now (or forever).
Looks good. Then we can upload files from the frontend website as well.
Edit : Found that we could not then send the other metadata with file if we just use -d flag. So, we will be forced to use multipartform content-type
In the last meeting, I discussed with @perazz , about the current issue with having a namespace token for uploading packages.
Currently, namespace maintainers can create tokens that are used for uploading the packages to the namespace. But, we also have a package maintainer role. Package maintainers are only responsible for maintaining packages (uploading new versions, adding other maintainers to the package).
But, currently, in order to get the token you have to be a namespace maintainer. Suppose, I add a person as a package maintainer thinking now they can be able to upload new versions of that package and now they want token from me because I am a namespace maintainer and hence they can't work independently and have to be dependent on the namespace maintainer for upload tokens which basically questions the role of package maintainer in itself.
So, I have a solution to allow package maintainers to generate tokens for the package itself and the token can be then used to upload a new version of that particular package.
For the fpm
, there will be no change as such required, we will be checking whether the token received is a namespace upload token or package specific token on the backend and send the correct response to the user.
@minhqdao @henilp105 @perazz What are your thoughts about this ?
In the last meeting, I discussed with @perazz , about the current issue with having a namespace token for uploading packages.
Currently, namespace maintainers can create tokens that are used for uploading the packages to the namespace. But, we also have a package maintainer role. Package maintainers are only responsible for maintaining packages (uploading new versions, adding other maintainers to the package).
But, currently, in order to get the token you have to be a namespace maintainer. Suppose, I add a person as a package maintainer thinking now they can be able to upload new versions of that package and now they want token from me because I am a namespace maintainer and hence they can't work independently and have to be dependent on the namespace maintainer for upload tokens which basically questions the role of package maintainer in itself.
So, I have a solution to allow package maintainers to generate tokens for the package itself and the token can be then used to upload a new version of that particular package.
For the
fpm
, there will be no change as such required, we will be checking whether the token received is a namespace upload token or package specific token on the backend and send the correct response to the user.@minhqdao @henilp105 @perazz What are your thoughts about this ?
Thanks, @arteevraina. I think that's perfect. A package maintainer should be able to generate tokens for that particular namespace + package, but not for other packages of that namespace.
Let's define the api for uploading packages. How do you want a JSON request for uploading a package to look like? @arteevraina @henilp105