Did it work?

[scripting]

See this Scripting News post. And thanks for your help testing feedToMasto! :smile:

[scotthansonde]

@scripting It worked. 😃 https://social.scottfr.ee/@scott got a whole bunch of new posts at once (since I set flOnlyPostNewItems to false so I could see immediate results).

[andysylvester]

I also got it to work (https://mastodon.social/@AndySylvester) using my FeedLand feed as the test feed.

Some feedback on the README (which was very good):

1. Recommend adding a sentence either at the end of the last section or beginning of this section to state that you should click on the application name in the list of applications (that action will show the keys needed in “Back in config.json section”.
2. Adding instructions about performing npm install and setting up forever if you want to keep the app running (like this example for my OPML includes demo (https://github.com/andysylvester/opml_includes_demo/blob/main/README.md)

Overall - pretty easy for me!

[scripting]

Thanks @andysylvester, @scotthansonde for giving it a test. It's a big relief that it worked! :smile:

I just released a new version with several changes.

1. Stripped markup from the description before sending it to Mastodon. They neuter the HTML, which is not a good look. Andy I noticed this in your tweets, could you install the new version and test to be sure it still works.

2. It checks feeds immediately at startup. There's no reason for it to wait a minute to do that.

3. I made one of the changes Andy recommended. There was indeed a step missing.

I did not add instructions for installing a Node app. Its omission was deliberate. I'm not trying to support "poets" anymore. That's why I say you have to have basic knowledge of running Node apps. People get the idea that they can daydream through running a server app and of course that doesn't work.

[scripting]

@scotthansonde -- you need the new version too...

https://social.scottfr.ee/@scott/109445346848129654

[scripting]

This is why we have testers! :-)

[andysylvester]

No worries - I think your path of asking for people with at least basic Node.js knowledge to test things of this type is the right path.

[andysylvester]

I will install the new version in a few minutes.

[andysylvester]

Looks good, not seeing markup in the posts now:

https://mastodon.social/@AndySylvester/109446302340310010

https://mastodon.social/@AndySylvester/109446306261447036

https://mastodon.social/@AndySylvester/109446310205782662

[bjimba]

OK, my first attempt.

I can get feedtomasto fired up, it spits the config file back at me, goes into the feed loop (quietly, before I started sticking in logs to see what's up). My config is two feeds from my blogger account (don't hate me for blogger, I'm inertial) - the main posts and the comments. When I post on blogger, my Thunderbird reader sees the post, my LifeRea sees the post, and from feedtomasto, I get a log message:

The request returned a status code of 302.

Nothing appears to have shown up at @jimr@fosstodon.org

[scripting]

Hi #bjimba --

302 is a redirect request. Not sure where it's coming from in your report. It would help if you told us what your feeds are in config.json.

[bjimba]

"feeds": [ "https://bjimba.blogspot.com/feeds/posts/default", "https://bjimba.blogspot.com/feeds/comments/default" ],

[scripting]

@bjimba -- the feeds look fine

I forgot to follow redirects, just added the code for that producing v0.4.3.

Could you download the new version and see if that fixes this problem?

[bjimba]

Well, it's a different error now.

Unexpected token < in JSON at position 0

[pzingg]

UPDATED (It works with Pleroma servers):

I tested against a locally running Pleroma-based Mastodon instance; I'm actually using a fork called Akkoma. I had to make a few tweaks to get things to work:

In tootStatus:

1. Added content_type: "text/html" ("text/markdown" also works) to the params object.
2. Moved params into the body argument for the mastopost call (using JSON.stringify).

In mastopost:

1. Added a Content-Type: application/json request header.
2. Only build the query string if params is not undefined.

Pleroma's API routing does not accept the status in the query string, only in the body of the HTTP POST. When creating a new status, the Pleroma API accepts a content_type param. The default if omitted is "text/plain". Here are the results for different content_type values:

1. "text/plain": New toots escape the HTML. Not good unless the original feed description has been stripped.
2. "text/html": Looks good, like the original feed item description. The title shows up as "### Title".
3. "text/markdown" (with "flServerSupportsMarkdown": true): Also looks good, but not identical to the original feed item.

[billstclair]

I tested it, posting the feed from my @billstclair@gleasonator.com to @iquited@impeccable.social.

It worked, but I had a couple of glitches.

1. I initially coped the "Bearer " prefix of the access token. Then I looked at the code, and noticed that it prepends that. I didn't mint my token according to your instructions. I let Mammudeck do it, and copied the token from LocalStorage. Note that this application doesn't use the clientKey, clientSecret, or scopes from config.json, just urlMastodonServer and accessToken.
2. It really should create the app and mint the token itself, but I have never done that from a stand-alone program, only from a web site. I'm pretty sure it's a standard OAuth use case. Asking regular users to create apps and tokens will be too much for many.

After that, it just worked, though the text in the created post appeared three times.

RSS Feed: https://gleasonator.com/users/billstclair/feed.rss Source post: https://gleasonator.com/@billstclair/posts/AQEwQY9MGuKBfZmjGS Created post: https://impeccable.social/notice/AQEwVwBgvepP6vTLbE

I note that images are not included, but since there's a link to the original, that probably isn't necessary.

[scripting]

@billstclair -- thanks for trying it, but I'm having trouble following tne narrative.

Right now my only concern is does it work if one follows the directions on the site.

Other concerns about what it shouldn't do -- I'd like to deal with those separately.

So it seems like there is at least one problem in the report -- "the text in the created post appeared three times."

Can we focus on that for a bit?

[scripting]

@pzingg -- I wrote this to work with Mastodon. I think that's pretty clear in the docs?

[scripting]

@bjimba -- do you have any idea what the app was trying to do when you got that error.

I tried loading your feeds into my copy but didn't get any errors.

Have you checked the syntax of your config.json file?

[billstclair]

Note that I also tested posting to a Pleroma server. That may be why I saw three copies of the text. So, if you're not going to support Pleroma, and you're going to require the user to create the app and token by hand, my feedback was not useful.

[scripting]

@billstclair -- I don't know anything about Pleroma. I'm very new to Mastodon, as I think you know.

However I am most definitely, for good reasons, requiring the user to create the app by hand. Mainly I don't like supporting helpless people. Or people with agendas. I was very clear up front about why I did this project.

[billstclair]

Pleroma, from what I've read, does not support as many users as Mastodon, but it has more (and, on average, smaller servers). Mastodon came first, then Pleroma, but those two are the backbone of the Fediverse. There are a whole bunch of other implementations, with the Mastodon Client API on one end and ActivityPub on the other, but those are the major two. I have found it easy to support both, with only a handful of cases where I need to something different depending on the server I'm talking to.

I have yet to see ANY Mastodon or Pleroma client that requires you to EVER look at the "</> Development" settings page. They all use the standard OAuth mechanism to create an app, and then then commands to create a code and use it to mint a token.

You've done the part to create the code and mint the token. The only part you haven't done is to create the app, which is normally how a Mastodon client gets a client_id and client_secret. But if you want to filter your users to those who can handle the "</> Developmet" page, that's your choice.

[scripting]

@billstclair -- please focus. Based on what you've said here i should get this so it works with Pieroma.

@pzingg has listed a set of steps that makes this work with Pieroma. Do you agree that this is the way to go?

[pzingg]

Sorry to open a can of worms about Pleroma.

[bjimba]

@davew - it's in function mastopost.

You only have 4 places where you call JSON.parse, so I added logs to see where I'm failing. It's the JSON.parse on line 173. At that point, "jsontext" contains HTML data.

This is the dump of "jsontext" before the call to JSON.parse:

<!doctype html>

[bjimba]

Warning -- this github commenting system appears to have stripped out some of what I pasted.

[billstclair]

Dave, your code worked without change to post to impeccable.social, except fot the three copies of the post text.

The following are POST /api/v1/statuses requests from Mammudeck, my Elm code, and my browser running the standard Pleroma web app (for Pleroma 1.0.6, a very old installation). I haven't added posting Markdown to Mammudeck yet. I assume it will work to just add a "content_type": "text/markdown" property to the JSON I'm already sending. I ALWAYS send JSON over the wire, only using URL parameters for the paging parameters (max_id, since_id, min_id, limit). I'll send you data from Gleasonator, if you want. It's running Soapbox/Rebased, a very new Pleroma fork.

I left out request headers that I thought were automatically generated, and probably some that I left in were also not put there by me.

Posting from Mammudeck (to impeccable.social, but it's server-independent): Resulted in: https://impeccable.social/notice/AQF5U3AmsxvJmfDyZk

Headers:

General
  Request URL: https://impeccable.social/api/v1/statuses
  Request Method: POST

Request Headers
  :authority: impeccable.social
  :method: POST
  :path: /api/v1/statuses
  :scheme: https
  accept: */*
  accept-encoding: gzip, deflate, br
  accept-language: en-US,en;q=0.9
  authorization: Bearer 076Sf...
  content-length: 96
  content-type: application/json
  user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36

Payload:
  {
  "status": "A simple post from Mammudeck, to see what goes over the wire.",
  "visibility": "public"
  }

---

Posting Markdown from the impeccable.social web site. Resulted in https://impeccable.social/notice/AQF51vkyricWfskFrU

Headers:

General
  Request URL: https://impeccable.social/api/v1/statuses
  Request Method: POST

Request Headers:
:authority: impeccable.social
:method: POST
:path: /api/v1/statuses
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.6
authorization: Bearer uMg2F...
content-length: 576
content-type: multipart/form-data; boundary=----WebKitFormBoundarypxXHzNajdwToHcBJ
cookie: __Host-pleroma_key=SFMyNTY.g3QAAAABbQAAAAd1c2VyX2lkbQAAABI5bVFUbzhXM1V2c2VsMkxvSjY.7uMOvrz9mSgCDhm0WuC6q-bhOunwTpwtNlBIXMgJDT8
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36

Payload:
  Form Data
    status: A `Markdown` post, to test what goes over the wire from by web browser to [impeccable.social](https://impeccable.social).
    source: Pleroma FE
    visibility: public
    content_type: text/markdown

[scripting]

@bjimba -- i have no idea what that code means or why whatever server you're talking to responded to the POST with that response.

@pzingg -- you did fine, esp now that i know that Pleroma is a commonly used server. i'm new to this space. trying to get something useful out and get back to working on my real software project.

i've gotta take a break. i'm way overloaded by the problems i'm being asked to deal with here. ;-)

[bjimba]

@davew - this is the contents of theRequest.url for the POST that is returning HTML

https://fostodon.org/api/v1/statuses?status=TEST%20POST%20-%20helping%20DaveW%20create%20feedToMasto%20-%20you%20should%2C%20too%0A%0A%23%23%23%20Testing%2016%2C%2017%2C%2018%0A%0AAgain.%20%40davew%40mastodon.social%20%23feedToMasto%0A%0A%0Ahttps%3A%2F%2Fbjimba.blogspot.com%2F2022%2F12%2Ftesting-16-17-18.html%0A

[bjimba]

And by the way, we're not asking you to deal with anything. You asked us to test it and report what we saw. :)

[billstclair]

I concur with @bjimba. Feel free to ignore what I say, if it's not in line with what you want to do. Just being me.

[scripting]

@pzingg -- I read your comment carefully.

1. You say the content-type of the request should be application/json, but don't say anything about how you send the body of the request (the filedata param). I'm assuming you added code to JSON.stringify it.

2. Do you think this calling format will also work with Mastodon, or did you customize it so that it will only work with the server you're using.

If you want me to support this -- we need to flesh this out.

[scripting]

BTW, I'm not ignoring what you say. I want your help. It may come out awkwardly, but I want to get to the bottom of the concern, and at the same time, when I say I hear you and I've made a decision, then let's move on.

Have another look at the reasons I'm doing this and dig in a little into those. This is meant to be example code as much as it's a utility.

If you feel this deserves a higher-level UI and you're ready to host the app, then go for it.

I'm very clear -- I'm not doing this to set up a utility for everyone. I've been down all these roads many many times, and I think I know where each of them goes. :smile:

And also remember that every bit of work I do here competes for my time with other work I should be doing elsewhere. I have two very large products I am responsible for. And a community of users for each of them. I've been neglecting them for the last few weeks, and I've got to get back to that work.

[pzingg]

Dave, yes, I did JSON.stringify(params) as the filedata argument, and undefined for the original params argument. Inside mastopost I look to see if params is undefined and if so, skip building a query string. I would love to test the code against mastodon-3.5.5.

Of course I tried to build it from source on my machine and there are the usual hassles (it requires openssl 2, and I have openssl 3 installed). Tomorrow I will try to make a docker build and see if using the stringified params in the body of the request will work. The Mastodon API docs say that parameters can be sent either as query string, WWW form data, or JSON, so it should work.

However, I don't think that the main mastodon repository supports the content_type param. Not sure if it would 400 if you include it or whether it would be ignored. So probably no support for markdown. It does look like glitch-soc, a popular fork of mastodon, would handle this.

[scripting]

Just released a new version with mastopost sending the params in the body of the post not as url params.

https://github.com/scripting/feedToMasto/blob/main/feedtomasto.js#L154

This is the way forms work, and posts are basically emulating forms, so it seemed this was the most conservative way to go and should give the maximum interop.

[scotthansonde]

I git pulled the new version, started it, added a post to my linkblog, it showed up on my test mastodon account https://social.scottfr.ee/@scott/109456282155944716

[pzingg]

New version with params in body works with both mastodon-3.5.5 and akkoma-2022.11! Also, adding content_type: "text/html" or content_type: "text/markdown" to the params works with both server types (does not error out in mastodon). I believe mastodon throws the content type away, while akkoma keeps the posted markdown content and content type in a "source" attribute of the status (toot).

[billstclair]

It worked for me, pulling from my post feed at gleasonator.com, and posting to impeccable.social, an ancient Pleroma instance, freeatlantis.com, an up-to-date Mastodon instance, and gleasonator.com, an up-to-date Soapbox/Rebased instance (fork of Pleroma).

[bjimba]

The 0.4.4 change did not help with my issue of fosstodon.org giving me HTML when JSON is expected. No time today for debugging, I'll let you know if I figure out anything.

[bjimba]

Well, I figured it out. Turns out that I had a typo in config.json (left out an 's' in fosstodon). The thing that made it tricky to find out is that someone has typo-squatted the single-s "fostodon.org" and was sending me a parking page.

I can now confirm that we're working with fosstodon.org and mastodon.online