Closed outaTiME closed 1 year ago
neet
commented
1 year ago @outaTiME Unfortunately, I couldn't reproduce the problem you're facing. The snippet you provided worked without any problem in my environment.
Since media uploading is one of the most complex layers that consists of factors depending on your environment, I cannot infer what is going on just by seeing the error messages. I would be able to identify the problem sooner if you show me a repository for the minimal reproduction.
neet
commented
1 year ago According to the log you provided, the media is uploaded in https://files.mastodon.social/media_attachments/files/109/706/560/687/655/487/original/9eac5d1990ac9dd7.jpeg
If you try to create a status with an unprocessed media ID, you'll get this error instead of attaching nothing.
file:///Users/nucx/ghq/github.com/neet/masto.js/dist/index.js:150
return new MastoHttpUnprocessableEntityError(message, props);
^MastoHttpUnprocessableEntityError: Cannot attach files that have not finished processing. Try again in a moment!
at createHttpError (file:///Users/nucx/ghq/github.com/neet/masto.js/dist/index.js:150:20)
at HttpNativeImpl.
- Interestingly, I tried to create a status in `mastodon.social` with a media attachment by specifying `"109706560687655487"` which is the ID you're claiming that you failed to create a status with, and it turned out that I've got a status with empty media. However, this never happens with my own media.
```ts
const { url } = await masto.v1.statuses.create({
status: 'caption',
visibility: 'private',
mediaIds: ['109706560687655487'],
});Outcome:
As the next step, I tried to download the image you uploaded by the following command
curl -g https://files.mastodon.social/media_attachments/files/109/706/560/687/655/487/original/9eac5d1990ac9dd7.jpeg > media.jpegAnd then upload it on my own
const file = fs.readFileSync('./sandbox/media.jpeg');
const masto = await login({
url: process.env.MASTODON_URL,
accessToken: process.env.MASTODON_TOKEN,
logLevel: 'debug',
});
const attachment = await masto.v2.mediaAttachments.create({ file });
const { url } = await masto.v1.statuses.create({
status: 'caption',
visibility: 'direct',
mediaIds: [attachment.id],
});This successfully created a post.
neet
commented
1 year ago @outaTiME I suspect some error occurred on the server side during the image processing. Can you try identifying the factor by yourself by changing the image to upload or instance?
If you still fail, it is highly possible that the problem is not caused by Masto.js and rather a problem in Mastodon itself or some other component that you're using.
I look forward to your reply.
outaTiME
commented
1 year ago Hi @neet, thanks for quick response, and I understand that it is not an easy case to reproduce given the complex infrastructure.
This is an open source project, and it is available online at: https://github.com/outaTiME/ambito-dolar, but unfortunately I can't commit my local changes yet.
The piece of code I shared with you works most of the time, but there are cases where it doesn't.
The script below uses a buffer generated from puppeteer (https://github.com/outaTiME/ambito-dolar/blob/master/packages/api/src/libs/chrome.js) which is used in parallel to upload an image to instagram and mastodon.
I could try generating a copy of the buffer to rule out that it is a concurrency problem in reading.
import AmbitoDolar from '@ambito-dolar/core';
import { generateScreenshot } from '../libs/chrome';
import { publish as publishToInstagram } from '../libs/instagram';
import { publish as publishToMastodon } from '../libs/mastodon';
import Shared from '../libs/shared';
export const handler = Shared.wrapHandler(async (event) => {
const {
type,
title = AmbitoDolar.getNotificationTitle(type),
caption,
} = JSON.parse(event.Records[0].Sns.Message);
// must required for social notification
if (!type || !title || !caption) {
throw new Error('Message is malformed, missing or has an invalid value');
}
console.info(
'Message received',
JSON.stringify({
type,
title,
caption,
})
);
const screenshot_url = Shared.getSocialScreenshotUrl({
title,
});
const promises = [];
try {
const {
target_url: image_url,
ig_file: file,
ig_story_file: story_file,
} = await generateScreenshot(screenshot_url);
promises.push(publishToInstagram(file, caption, story_file));
promises.push(publishToMastodon(caption, file));
promises.push(
Shared.triggerSendSocialNotificationsEvent(caption, image_url)
);
} catch (error) {
console.warn(
'Unable to generate the screenshot for notification',
JSON.stringify({ type, title, error: error.message })
);
// send as plain
}
const results = await Promise.all(promises);
console.info('Completed', JSON.stringify(results));
return results;
});But again, which is weird to me (as I mentioned above) the image to mastodon would seem to upload correctly.
Another thing I wanted to do was try the 4.x version of masto.js to see if it works without problems with axios.
neet
commented
1 year ago @outaTiME Thank you for providing the repo. I'll try running it later in my local environment and see what is wrong.
outaTiME
commented
1 year ago thanks again @neet , I'll keep testing and keep you informed
outaTiME
commented
1 year ago @neet today I had a double case again,
in the first execution of the service (at the beginning of the day 10:00 UTC) the upload of the image was done without errors or retries and it correctly impacted the creation of the status and in the second execution ( at the end of the day 21:00 UTC) throw HTTP failed Response error twice and after that the image uploaded successfully, but the status was created without it.
The strange thing is that if there are errors in the debug log (cause of the waitFor because the upload takes time to hit) the state creation is done without attachments.
Then I leave both raw logs to see if they are useful.
neet
commented
1 year ago @outaTiME I'm glad to hear that the bot at least worked once. Let's make it work constantly 💪
HTTP failed Response
As far as I know, some instances including mastodon.social caches their API response at a specific layer. As a result, they return 404 for a media attachment which has just been created for the time being. This is totally expected so you won't see this error unless you explicitly turn logLevel: "debug" on.
This is why 404 errors are logged to your console several times, and v2.mediaAttachments.create (internally calls waitFor) automatically waits for url property of the media attachment to be truthy for you ― because you can't create a status during the process
The strange thing is that if there are errors in the debug log (cause of the waitFor because the upload takes time to hit) the state creation is done without attachments.
Interesting. I suppose this might because of the cache layer. What happens if you put a sleep in between mediaAttachments.create and statuses.create?
outaTiME
commented
1 year ago @neet Since you answered me, I have been testing, and I have not had any other HTTP failed Response error, possibly something from mastodon.social as you say, for now everything seems to work normally.
I will continue testing at other times, and I also left a sleep of 5 seconds in my service.
outaTiME
commented
1 year ago Today, after a long time, I once again had a failure to publish the status with an image, it could really be a problem with the mastodon.social server or some condition that causes this.
One thing that never ceases to amaze me is that after the retries, the image is uploaded, but not along with the status.
Unfortunately, I don't think my contribution will be of much use, but I am leaving the execution logs.
outaTiME
commented
1 year ago A case from today, the upload was fine with no errors or retries, but the status was uploaded without the image, maybe it's definitely something from the mastodon.social server.
Analyzing a case with errors and retries from last Friday, the status went up without an image, but today looking at my user's timeline I see that the image appears (when days ago it didn't), I still don't understand clearly what it is It does happen, but again it would appear to be something from the mastodon.social server.
outaTiME
commented
1 year ago Hi there, I wanted to tell you that since a long time ago the image upload is working consistently, it seems that it was something of mastodon server side. No need to keep this issue open, greetings, and thanks for the support 🫶
I've been debugging this problem for several days now and I don't understand the reason for it. Randomly, the attachments upload correctly (apparently), but they don't save along with my status creation.
I have a service running in AWS on Node 16 with version 5.5.1 of masto and configuring the library with debug mode enabled, I see a 404 (HTTP failed Response) when getting the image and then in a subsequent call like that request could be resolved (possibly this is due to the
waitForlogic), however the call to create the status is made without the attachment (blankmedia_attachments) even though it has been uploaded correctly.This is the code that makes the creation (the
filethat comes as a parameter is a buffer):And here I leave the logs generated by the service:
Click to see full log
``` INFO ↑ GET https://mastodon.social/api/v1/instance undefined DEBUG body null INFO ↓ GET https://mastodon.social/api/v1/instance undefined DEBUG body {""uri"":""mastodon.social"",""title"":""Mastodon"",""short_description"":""The original server operated by the Mastodon gGmbH non-profit"",""description"":"""",""email"":""staff@mastodon.social"",""version"":""4.0.2"",""urls"":{""streaming_api"":""wss://mastodon.social""},""stats"":{""user_count"":921486,""status_count"":48627565,""domain_count"":48720},""thumbnail"":""https://files.mastodon.social/site_uploads/files/000/000/001/@1x/57c12f441d083cde.png"",""languages"":[""en""],""registrations"":false,""approval_required"":false,""invites_enabled"":true,""configuration"":{""accounts"":{""max_featured_tags"":10},""statuses"":{""max_characters"":500,""max_media_attachments"":4,""characters_reserved_per_url"":23},""media_attachments"":{""supported_mime_types"":[""image/jpeg"",""image/png"",""image/gif"",""image/heic"",""image/heif"",""image/webp"",""image/avif"",""video/webm"",""video/mp4"",""video/quicktime"",""video/ogg"",""audio/wave"",""audio/wav"",""audio/x-wav"",""audio/x-pn-wave"",""audio/vnd.wave"",""audio/ogg"",""audio/vorbis"",""audio/mpeg"",""audio/mp3"",""audio/webm"",""audio/flac"",""audio/aac"",""audio/m4a"",""audio/x-m4a"",""audio/mp4"",""audio/3gpp"",""video/x-ms-asf""],""image_size_limit"":10485760,""image_matrix_limit"":16777216,""video_size_limit"":41943040,""video_frame_rate_limit"":60,""video_matrix_limit"":2304000},""polls"":{""max_options"":4,""max_characters_per_option"":50,""min_expiration"":300,""max_expiration"":2629746}},""contact_account"":{""id"":""1"",""username"":""Gargron"",""acct"":""Gargron"",""display_name"":""Eugen Rochko"",""locked"":false,""bot"":false,""discoverable"":true,""group"":false,""created_at"":""2016-03-16T00:00:00.000Z"",""note"":""\u003cp\u003eFounder, CEO and lead developer \u003cspan class=\""h-card\""\u003e\u003ca href=\""https://mastodon.social/@Mastodon\"" class=\""u-url mention\""\u003e@\u003cspan\u003eMastodon\u003c/span\u003e\u003c/a\u003e\u003c/span\u003e, Germany.\u003c/p\u003e"",""url"":""https://mastodon.social/@Gargron"",""avatar"":""https://files.mastodon.social/accounts/avatars/000/000/001/original/dc4286ceb8fab734.jpg"",""avatar_static"":""https://files.mastodon.social/accounts/avatars/000/000/001/original/dc4286ceb8fab734.jpg"",""header"":""https://files.mastodon.social/accounts/headers/000/000/001/original/3b91c9965d00888b.jpeg"",""header_static"":""https://files.mastodon.social/accounts/headers/000/000/001/original/3b91c9965d00888b.jpeg"",""followers_count"":288544,""following_count"":342,""statuses_count"":73066,""last_status_at"":""2023-01-17"",""noindex"":false,""emojis"":[],""fields"":[{""name"":""Patreon"",""value"":""\u003ca href=\""https://www.patreon.com/mastodon\"" target=\""_blank\"" rel=\""nofollow noopener noreferrer me\""\u003e\u003cspan class=\""invisible\""\u003ehttps://www.\u003c/span\u003e\u003cspan class=\""\""\u003epatreon.com/mastodon\u003c/span\u003e\u003cspan class=\""invisible\""\u003e\u003c/span\u003e\u003c/a\u003e"",""verified_at"":null}]},""rules"":[{""id"":""1"",""text"":""Sexually explicit or violent media must be marked as sensitive when posting""},{""id"":""2"",""text"":""No racism, sexism, homophobia, transphobia, xenophobia, or casteism""},{""id"":""3"",""text"":""No incitement of violence or promotion of violent ideologies""},{""id"":""4"",""text"":""No harassment, dogpiling or doxxing of other users""},{""id"":""5"",""text"":""No content illegal in Germany""},{""id"":""7"",""text"":""Do not share intentionally false or misleading information""}]} DEBUG Masto.js initialised MastoConfig { props: { url: 'https://mastodon.social', accessToken: 'xxx', logLevel: 'debug', version: SemVer2 { options: [Object], loose: true, includePrerelease: false, raw: '4.0.2', major: 4, minor: 0, patch: 2, prerelease: [], build: [], version: '4.0.2' }, streamingApiUrl: 'wss://mastodon.social' }, serializer: SerializerNativeImpl {} } INFO ↑ POST https://mastodon.social/api/v2/media undefined DEBUG body FormData3 { _overheadLength: 160, _valueLength: 507717, _valuesToMeasure: [], writable: false, readable: true, dataSize: 0, maxDataSize: 2097152, pauseStreams: true, _released: false, _streams: [ '----------------------------793820681180656593245372\r ' + 'Content-Disposition: form-data; name=""file""; filename=""blob""\r ' + 'Content-Type: application/octet-stream\r ' + '\r ',