The social web translator. Fetches and converts data between social networks, HTML and JSON with microformats2, ActivityStreams/ActivityPub, Atom, JSON Feed, and more.
Granary is a library and REST API that fetches and converts between a wide variety of social data sources and formats:
Free yourself from silo API chaff and expose the sweet social data foodstuff inside in standard formats and protocols!
Here's how to get started:
pip install granary
.License: This project is placed in the public domain. You may also use it under the CC0 License.
The library and REST API are both based on the OpenSocial Activity Streams service. Let's start with an example. This code using the library:
from granary import twitter
...
tw = twitter.Twitter(ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET)
tw.get_activities(group_id='@friends')
is equivalent to this HTTP GET
request:
https://granary.io/twitter/@me/@friends/@app/
?access_token_key=ACCESS_TOKEN_KEY&access_token_secret=ACCESS_TOKEN_SECRET
They return the authenticated user's Twitter stream, ie tweets from the people they follow. Here's the JSON output:
{
"itemsPerPage": 10,
"startIndex": 0,
"totalResults": 12,
"items": [{
"verb": "post",
"id": "tag:twitter.com,2013:374272979578150912",
"url": "http://twitter.com/evanpro/status/374272979578150912",
"content": "Getting stuff for barbecue tomorrow. No ribs left! Got some nice tenderloin though. (@ Metro Plus Famille Lemay) http://t.co/b2PLgiLJwP",
"actor": {
"username": "evanpro",
"displayName": "Evan Prodromou",
"description": "Prospector.",
"url": "http://twitter.com/evanpro",
},
"object": {
"tags": [{
"url": "http://4sq.com/1cw5vf6",
"startIndex": 113,
"length": 22,
"objectType": "article"
}, "..."],
},
}, "..."]
"..."
}
The request parameters are the same for both, all optional: USER_ID
is a source-specific id or @me
for the authenticated user. GROUP_ID
may be @all
, @friends
(currently identical to @all
), @self
, @search
, or @blocks
; APP_ID
is currently ignored; best practice is to use @app
as a placeholder.
Paging is supported via the startIndex
and count
parameters. They're self explanatory, and described in detail in the OpenSearch spec and OpenSocial spec.
When using the GROUP_ID
@search
(for platforms that support it — currently Twitter and Instagram), provide a search string via the q
parameter. The API is loosely based on the OpenSearch spec, the OpenSocial Core Container spec, and the OpenSocial Core Gadget spec.
Output data is JSON Activity Streams 1.0 objects wrapped in the OpenSocial envelope, which puts the activities in the top-level items
field as a list and adds the itemsPerPage
, totalCount
, etc. fields.
Most Facebook requests and all Twitter, Instagram, and Flickr requests will need OAuth access tokens. If you're using Python on Google App Engine, oauth-dropins is an easy way to add OAuth client flows for these sites. Otherwise, here are the sites' authentication docs: Facebook, Flickr, Instagram, Twitter.
If you get an access token and pass it along, it will be used to sign and authorize the underlying requests to the sources providers. See the demos on the REST API endpoints above for examples.
The endpoints above all serve the OpenSocial Activity Streams REST API. Request paths are of the form:
/USER_ID/GROUP_ID/APP_ID/ACTIVITY_ID?startIndex=...&count=...&format=FORMAT&access_token=...
All query parameters are optional. FORMAT
may be as1
(the default), as2
, atom
, html
, jsonfeed
, mf2-json
, rss
, or xml
(the default). atom
supports a boolean reader
query parameter for toggling rendering appropriate to feed readers, e.g. location is rendered in content when reader=true
(the default). The rest of the path elements and query params are described above.
Errors are returned with the appropriate HTTP response code, e.g. 403 for Unauthorized, with details in the response body.
By default, responses are cached and reused for 10m without re-fetching the source data. (Instagram responses are cached for 60m.) You can prevent this by adding the cache=false
query parameter to your request.
Include the shares=false
query parameter to omit shares, eg Twitter retweets, from the results.
To use the REST API in an existing ActivityStreams/ActivityPub client, you'll need to hard-code exceptions for the domains you want to use e.g. facebook.com
, and redirect HTTP requests to the corresponding endpoint above.
Facebook and Instagram are disabled in the REST API entirely, sadly.
See the example above for a quick start guide.
Clone or download this repo into a directory named granary
. Each source works the same way. Import the module for the source you want to use, then instantiate its class by passing the HTTP handler object. The handler should have a request
attribute for the current HTTP request.
The useful methods are get_activities()
and get_actor()
, which returns the current authenticated user (if any). See the full reference docs for details. All return values are Python dicts of decoded ActivityStreams 1 JSON.
The microformats2.*_to_html()
functions are also useful for rendering ActivityStreams 1 objects as nicely formatted HTML.
Check out the oauth-dropins Troubleshooting/FAQ section. It's pretty comprehensive and applies to this project too.
We'd love to add more sites! Off the top of my head, YouTube, Tumblr, WordPress.com, Sina Weibo, Qzone, and RenRen would be good candidates. If you're looking to get started, implementing a new site is a good place to start. It's pretty self contained and the existing sites are good examples to follow, but it's a decent amount of work, so you'll be familiar with the whole project by the end.
Pull requests are welcome! Feel free to ping me in #indieweb-dev with any questions.
First, fork and clone this repo. Then, install the Google Cloud SDK and run gcloud components install cloud-firestore-emulator
to install the Firestore emulator. Once you have them, set up your environment by running these commands in the repo root directory:
gcloud config set project granary-demo
python3 -m venv local
source local/bin/activate
pip install -r requirements.txt
# needed to serve static files locally
ln -s local/lib/python3*/site-packages/oauth_dropins/static oauth_dropins_static
Now, run the tests to check that everything is set up ok:
gcloud emulators firestore start --host-port=:8089 --database-mode=datastore-mode < /dev/null >& /dev/null &
python3 -m unittest discover
Finally, run the web app locally with flask run
:
GAE_ENV=localdev FLASK_ENV=development flask run -p 8080
Open localhost:8080 and you should see the granary home page!
If you want to work on oauth-dropins at the same time, install it in editable mode with pip install -e <path to oauth-dropins repo>
. You'll also need to update the oauth_dropins_static
symlink, which is needed for serving static file handlers locally: ln -sf <path-to-oauth-dropins-repo>/oauth_dropins/static oauth_dropins_static
.
To deploy to production:
gcloud -q beta app deploy --no-cache granary-demo *.yaml
The docs are built with Sphinx, including apidoc, autodoc, and napoleon. Configuration is in docs/conf.py
To build them, first install Sphinx with pip install sphinx
. (You may want to do this outside your virtualenv; if so, you'll need to reconfigure it to see system packages with virtualenv --system-site-packages local
.) Then, run docs/build.sh
.
Here's how to package, test, and ship a new release. (Note that this is largely duplicated in the oauth-dropins readme too.)
source local/bin/activate.csh
CLOUDSDK_CORE_PROJECT=granary-demo gcloud emulators firestore start --host-port=:8089 --database-mode=datastore-mode < /dev/null >& /dev/null &
sleep 5
python -m unittest discover
kill %1
deactivate
setup.py
and docs/conf.py
. git grep
the old version number to make sure it only appears in the changelog. Change the current changelog entry in README.md
for this new version from unreleased to the current date.oauth-dropins
version specifier in setup.py
to the most recent version.docs/source/
. Then run ./docs/build.sh
. Check that the generated HTML looks fine by opening docs/_build/html/index.html
and looking around.git commit -am 'release vX.Y'
python setup.py clean build sdist
setenv ver X.Y
source local/bin/activate.csh
twine upload -r pypitest dist/granary-$ver.tar.gz
cd /tmp
python -m venv local
source local/bin/activate.csh
pip uninstall granary # make sure we force Pip to use the uploaded version
pip install --upgrade pip
pip install mf2py==1.1.2
pip install -i https://test.pypi.org/simple --extra-index-url https://pypi.org/simple granary==$ver
deactivate
Smoke test that the code trivially loads and runs.
source local/bin/activate.csh
python
# run test code below
deactivate
Test code to paste into the interpreter:
import json
from granary import github
github.__file__ # check that it's in the virtualenv
g = github.GitHub('XXX') # insert a GitHub personal OAuth access token
a = g.get_activities()
print(json.dumps(a, indent=2))
from granary import atom
print(atom.activities_to_atom(a, {}))
### Notable changes
on the second line, then copy and paste this version's changelog contents below it.
git tag -a v$ver --cleanup=verbatim
git push && git push --tags
vX.Y
in the Tag version box. Leave Release title empty. Copy ### Notable changes
and the changelog contents into the description text box.twine upload dist/granary-$ver.tar.gz
Apache Streams is a similar project that translates between storage systems and database as well as social schemas. It's a Java library, and its design is heavily structured. Here's the list of formats it supports. It's mainly used by People Pattern.
Gnip similarly converts social network data to ActivityStreams and supports many more source networks. Unfortunately, it's commercial, there's no free trial or self-serve signup, and plans start at $500.
DataSift looks like broadly the same thing, except they offer self-serve, pay as you go billing, and they use their own proprietary output format instead of ActivityStreams. They're also aimed more at data mining as opposed to individual user access.
Cliqset's FeedProxy used to do this kind of format translation, but unfortunately it and Cliqset died.
Facebook used to officially support ActivityStreams, but that's also dead.
There are a number of products that download your social network data, normalize it, and let you query and visualize it. SocialSafe is one, although the SSL certificate is currently out of date. ThinkUp was an open source product, but shuttered on 18 July 2016. There's also the lifelogging/lifestream aggregator vein of projects that pull data from multiple source sites. Storytlr is a good example. It doesn't include Facebook, or Instagram, but does include a number of smaller source sites. There are lots of others, e.g. the Lifestream WordPress plugin. Unfortunately, these are generally aimed at end users, not developers, and don't usually expose libraries or REST APIs.
On the open source side, there are many related projects. php-mf2-shim adds microformats2 to Facebook and Twitter's raw HTML. sockethub is a similar "polyglot" approach, but more focused on writing than reading.
Standardize function and method names in all modules to to_as1
, from_as
, etc. Old method names are now deprecated but won't be removed until at least v8.0, if not later.
as1
:
is_dm
, recipient_if_dm
, get_id
, and is_audience
functions.as2
:
sensitive
, indexable
, and discoverable
support.is_server_actor
function (FEP-d556, discussion).from_as1
:type: Image
, never to bare string URLs (bridgy-fed#/1000).to_as1
:Hashtag
and inner tag
field for name.mimeType
goes in outer object, not in stream
.to
/cc
with mixed dict and string elements.link_tags
: add class="mention"
for Mention
tags (bridgy-fed/#887).atom
:
atom_to_activity/ies
: Get URL from link
for activities as well as objects. (Thanks @imax9000!)bluesky
:
app.bsky.feed.post#langs
to/from AS1 contentMap
(which isn't officially part of AS1; we steal it from AS2).sensitive
on posts to Bluesky graphic-media
self label, and many Bluesky self labels back to sensitive
with content warning(s) in summary
.create
/previewCreate
:inReplyTo
isn't a Bluesky URL or AT URI, return CreationResult
instead of raising ValueError
.from_as1
:article
s to external embeds with no post text.as_embed
boolean kwarg to do the same thing for any object.id
if url
is not available (snarfed/bridgy-fed#1155).inReplyTo
or object
or target
with no recognizable ATProto or Bluesky object, raise ValueError
.blobs
.flag
has multiple objects, use the first one that's an ATProto record.uri
s.title
in content
correctly.id
or url
.to_as1
:app.bsky.actor.profile#description
into url
/urls
fieldsurl
, list of URLs goes in urls
.ref
s as well as CID
instances.Bluesky.get_activities
: skip unknown record types instead of raising ValueError
.microformats2
:
object_to_json
: Improve handling of items with multiple types by removing inReplyTo
from likes, shares, etc (snarfed/bridgy-fed#941).rss
:
Breaking changes:
jsonfeed
:
jsonfeed_to_activities
: return AS1 objects, not activities.Non-breaking changes:
as1
:
activity_changed
: add displayName
, summary
fields.is_public
: return False
if the object/activity contains to
that's empty or has only unknown aliases.as2
:
Application
, Block
, Flag
, and Link
types.to/from_as1
across all actor types, not just Person
.link_tags
function.atom
:
activities_to_atom
: handle image attachments without url
field.bluesky
:
to_as1
:app.bsky.embed.record
app.bsky.embed.recordWithMedia
app.bsky.feed.defs#notFoundPost
app.bsky.feed.generator
app.bsky.graph.block
app.bsky.graph.list
app.bsky.graph.listitem
com.atproto.admin.defs#repoRef
com.atproto.moderation.createReport#input
com.atproto.repo.strongRef
getBlob
image URLs.app.bsky.actor.profile
: add HTML links for URLs in summary
(snarfed/bridgy-fed#1065).<
, >
, &
) in app.bsky.actor.profile
description
field.create
/update
activities with bare string object
.from_as1
:tags
with missing objectType
as hashtags.maxGraphemes
or maxLength
in its lexicon, truncate it with an …
ellipsis character at the end in order to fit. If this happens to post text, include a link embed pointing to the original post.[Video]
(snarfed/bridgy-fed#1078).note
has summary
- often used for content warnings in the fediverse - add it to content
as a prefix instead of overriding content
(snarfed/bridgy-fed#1001).reply.root
properly in reply posts (snarfed/bridgy#1696).original_fields_prefix
kwarg to store original data in custom (off-Lexicon) *OriginalDescription
and *OriginalUrl
fields in app.bsky.actor.profile
and *OriginalText
and *OriginalUrl
fields in app.bsky.feed.post
(snarfed/bridgy-fed#1092).lexrpc.Client
as well as Bluesky
for client
kwarg.from_as1_to_strong_ref
:value
boolean kwarg.client
kwarg from Bluesky
to lexrpc.Client
.microformats2
:
person
.json_to_object
:#
prefix (if present) from hashtag u-category
s.name
property is an object, eg an h-card
.object_to_json
:id
and url
inside inReplyTo
to in-reply-to.
nostr
:
source
:
Source.postprocess
: when extracting @-mentions, defer to existing tag if it has the same displayName
and has url
.as1
:
get_owner
bug fix for post
, update
, delete
activities.activity_changed
: add new inReplyTo
kwarg.is_public
: add new unlisted
kwarg.as2
:
to_as1
: bug fix, preserve objectType: featured
for banner/header images even when mediaType
is also set.is_public
: add new unlisted
kwarg.from_as1
:icon
field, prefer image types that are allowed by Mastodon.stop-following
with string object
id.atom
:
extract_entries
function.activity_to_atom
: default actor/author name to username.atom_to_activities
: support top-level entry
element as well as feed
.atom_to_*
:object.author
objectType
to article
/note
and verb
to post
link rel=self
/alternate
to url
displayName
in objects instead of title
link
without rel
as self link.entry.author
doesn't have id or url, default them to feed author's.bluesky
:
create
and preview
.record
and object
types in from_as1
and to_as1
. Use to_as1
's type
kwarg and from_as1
's out_type
kwarg to disambiguate.Bluesky.post_id
.blob_to_url
function.as1_to_profile
, switch from_as1
to return $type: app.bsky.actor.profile
.summary
and content
to plain text.Bluesky.user_to_actor
, Bluesky.get_actor
.at_uri_to_web_url
: support lists.web_url_to_at_uri
: convert profile URLs like https://bsky.app/profile/snarfed.org
to profile record URIs (at://snarfed.org/app.bsky.actor.profile/self
) instead of repo URIs (at://snarfed.org
).from_as1_to_strong_ref
.:
s in record keys (atproto#2224).to_as1
:getBlob
URLs.uri
kwarg.handle
to username
, add new repo_handle
kwarg.app.bsky.feed.repost
, app.bsky.graph.defs#listView
, app.bsky.feed.defs#blockedPost
.actor
/author
based on repo_did
.url
field: include custom handles, only use repo_did/handle
for app.bsky.actor.profile
.!no-unauthenticated
label on profiles to AS1 @unlisted
audience target (bridgy-fed#828).from_as1
:out_type
kwarg to specify desired output type, eg app.bsky.actor.profile
vs app.bsky.actor.defs#profileViewBasic
vs app.bsky.actor.defs#profileView
.blobs
kwarg to provide blob objects to use for image URLs.client
kwarg to fetch and populate CIDs.parent
as root
in replies. (Technically wrong in cases where the parent isn't the root, but we don't actually know the root. 🤷)image
field.url
field./
from rel-me verified links on Mastodon etc.attributedTo
to singular if it has only one element.name
isn't set, fall back to preferredUsername
or infer Webfinger handle from id
or url
.url
field (bridgy#1640).bsky.app
inReplyTo
URLs to at://
URIs.datetime
conversion to match the ATProto recommended format.facebook
:
Facebook.fql_stream_to_post
. Facebook turned down FQL in 2016.github
:
displayName
in objects instead of title
.mastodon
:
get_activities
bug fix: use query params for /api/v1/notifications
API call, not JSON body.error
JSON field (eg from Sharkey) to 400/401 exceptions.media_attachments.remote_url
when available since it may be more long-lived than url
for remote statuses (bridgy#1675).microformats2
:
object_to_json
bug fix: handle singular inReplyTo
.json_to_object
bug fix: handle list-valued location
.nostr:
get_*
: return partial results when the websocket connection is closed prematurely.to_as1
: handle invalid NIP05 values (eg {}
)rss
:
to_activities
:objectType: note
if title
isn't set or is a prefix (possibly ellipsized) of content
/description
.media:content
tags (#674).Source
:
postprocess_activity/object
: add mentions
kwarg to convert @-mentions in HTML links to mention
tags.Highlights: Nostr, Bluesky get_activities
, lots of improvements in as2
and microformats2
, and more!
REST API breaking changes:
Twitter is dead, at least in the REST API.
Non-breaking changes:
nostr
module!as1
:
get_owner
, targets
.accept
, reject
, stop-following
to VERBS_WITH_OBJECT
and remove repost
, it's not an AS1 verb.url
field list values (even though it's invalid AS1).as2
:
to_as1
:Video
handling: support Link
objects in url
, extract stream URLs and types from link tag
s.latitude
and longitude
to float, raise ValueError
on failure.image
as well as attachments
(bridgy-fed#429).value
s.mediaType
in attachment
and tags
.TYPES_WITH_OBJECT
constant.get_urls
, address
functions.Content-Type
compatibility with application/ld+json; profile="https://www.w3.org/ns/activitystreams"
.Undo
activities with bare string id object
s.PropertyValue
attachments on actors to include full URL in anchro text to be compatible with Mastodon's profile link verification.atom
:
activities_to_atom
etc:content
from XHTML to HTML inside CDATA to support non-XHTML input content (bridgy-fed#624.image
values.type="application/atom+xml"
from rel="self"
link
in entry
.objectType: comment
attachments.<a>
element for tags.<
and >
characters in title
(#629).activity_to_atom
/activities_to_atom
for dict-valued url
fields.objectType: service
attachments, eg Bluesky custom feeds.bluesky
:
Bluesky
API class, including get_activities
.app.bsky
/com.atproto
lexicons, use lexrpc's instead.web_url_to_at_uri
function.from_as1
: handle link tags without start/end indices.to_as1
:type
kwarg.did
into actor id
.app.bsky.feed.defs#generatorView
.as1_to_profile
.subject
in app.bsky.graph.follow
is followee, not follower. (That field is badly named!)jsonfeed
:
activities_to_jsonfeed
:image
and stream
.author
.mastodon
:
status_to_object
: add/fix alt text handling for images.microformats2
:
json_to_html
:json_to_object
:published
and updated
timestamps.object_to_json
:image
values.published
and updated
timestamps.replies
and shares
(usually from AS2.)render_content
:author
and actor
values.objectType: service
attachments, eg Bluesky custom feeds, in JSON and HTML output.rss
:
from_activities
: handle bare string id author
.Breaking changes:
as2
:
object
, inReplyTo
, etc values as ids, convert them to bare strings or id
instead of url
.microformats2
:
in-reply-to
, repost-of
, like-of
etc values to AS1 bare strings or id
s instead of url
s.Non-breaking changes:
bluesky
module for Bluesky/AT Protocol!as1
:
organization
object type and ACTOR_TYPES
constant (based on AS2).get_ids
, get_object
, and get_objects
functions.activity_changed
: ignore inReplyTo.author
(snarfed/bridgy#1338)as2
:
stop-following
and AS2 Undo
Follow
.Accept
and Reject
for follows as well as event RSVPs.Question
(ie poll), Organization
, and Delete
object types.to
/cc
to/from AS1 to
for public and unlisted.type: Document
video attachments like Mastodon emits.from_as1
: bug fix for image objects with url
and value
fields (for alt text).from_as1
: bug fix, handle bare string URL image
values.from_as1
: convert urls.displayName
to attachment.name
(bridgy-fed#331).from_as1
: preserve inReplyTo
object values as objects, inline single-element lists down down to just single element.to_as1
: use objectType: featured
for first image in image
field.to_as1
: populate actor
into object.author
for Update
s as well as Create
s.to_as1
: convert Mastodon profile metadata PropertyValue
attachments to url
composite objects with displayName
.to
and cc
values when converting both directions.atom
:
image
field to Atom.published
and updated
in entries with objects, eg likes, reposts, RSVPs, bookmarks. Thanks @gregorlove! (#480)activity/ies_to_atom
when object
is present and empty.objectType
in the to
field.flickr
:
get_activities
: add support for the count
kwarg.github
:
get_activities
: add support for the count
kwarg.jsonfeed
:
white-space: pre
CSS to converting newlines to <br>
s because some feed readers follow it strictly and don't even line wrap (#456).mastodon
:
microformats2
:
json_to_object
: drop backward compatibility support for like
and repost
properties. Background discussion.json_to_object
: add new rel_urls
kwarg to allow attaching displayName
s to urls
based on HTML text or title
attribute (bridgy-fed#331).json_to_activities
function.hcard_to_html
/maybe_linked_name
: when name
is missing, use pretty URL as visible text.h-card
org
property.json_to_object
: handle composite rsvp
property value.json_to_object
: bug fix when fetch_mf2
is True, handle when we run the authorship algorithm and fetch an author URL that has a u-photo
with alt
.rss
:
from_activities
: fix item ordering to match input activities.Breaking changes:
get_activities
cache
kwarg's support for App Engine memcache interface. It's now only used as a plain dict
. get_activities
will now make many small modifications, so if you pass an object that implements those as API calls, you'll probably want to batch those separately.create
/preview
: support the AS1 favorite
verb as well as like
. (bridgy#1345)id
(instead of url
) to Atom id
.get_actor
.create
/preview
: allow non-Mastodon replies, ie activities that include inReplyTo
URLs even if none of them point to a toot. (bridgy#1321)requests.HTTPError
with response.status_code
502 instead of JSONDecodeError
on non-JSON responses. This is synthetic, but more helpful for error handling.object_to_json
and related functions: handle all escaped HTML entities, not just &
<
>
.microformats2.prefix_image_urls
and prefix_video_urls
into a new as1.prefix_urls
function.itunes:category
. It has to be one of Apple's explicit categories, which we aren't prepared to validate, so don't try.url
and urls
from AS1 into multi-valued AS2 url
field.Source
class to a new as1
module: object_type
, merge_by_id
, is_public
, add_rsvps_to_event
, get_rsvps_from_event
, activity_changed
, append_in_reply_to
, actor_name
, original_post_discovery
.as1.original_post_discovery
: remove deprecated cache
kwarg.Non-breaking changes:
icon
and image
are singly valued, not multiply valued.is_public
method and PUBLIC_AUDIENCE
constant."objectType": "featured"
first in the image
field when converting from AS1, last in the icon
field. This matches the ActivityPub (Mastodon) convention of using icon
for profile pictures and image
for header images.url
values into new PropertyValue
attachments on Person
objects; these end up in Mastodon's "profile metadata" link fields.to_as1
: if an attachment's mediaType
is image/...
, override objectType
and set it to image
.data-ft
attribute and _ft_
query param more often instead of story_fbid
, which is now an opaque token that changes regularly. (facebook-atom#27)Instagram.scraped_json_to_activities
method.create
and preview
: convert profile URLs to @-mentions, eg https://github.com/snarfed
to @snarfed
(bridgy#1090).get_activities
with activity_id
now supports fetch_replies
and fetch_likes
.cache
support to get_activities
./scraped
endpoint that accepts POST
requests with silo HTML as input. Currently only supports Instagram. Requires site=instagram
, output=...
(any supported output format), and HTML as either raw request body or MIME multipart encoded file in the input
parameter.extra
and body_class
kwargs to activities_to_html
.u-featured
images to AS1, add new non-standard "objectType": "featured"
field to distinguish them from u-photo
.p-note
to AS1 summary
.image
attachments to photo
.Source.original_post_discovery
: add new max_redirect_fetches
keyword arg.Breaking changes:
Non-breaking changes:
rss.to_activities
function.include_shares
kwarg to get_activities
, implemented for Twitter and Mastodon. Defaults to True
. If False
, shares (retweets in Twitter, boosts in Mastodon) will be discarded and not returned. Also add a corresponding shares
query param to the REST API.user
object, add new fetch for comments.Instagram.merge_scraped_comments()
.type
isn't a string.get_activities()
to fetch posts by the current user or a user specified with user_id
.log_html
kwarg to get_activities
; defaults to False.items.author
element.Source.original_post_discovery
: add new include_reserved_hosts
kwarg, defaults to True
.feed_v2
JSON format.get_activities()
with fetch_mentions=True
: handle notifications with status: null
. Maybe happens when a status is deleted?create
/preview_create
: support bookmarks. (Nothing special happens with them; their content
is posted as a normal toot.)image.displayName
as visible text in HTML, since it's already in the <img>
's alt
attribute.bookmark-of
support.prefix_image_urls()
function.content
in AS1/2 objects.json_to_object
bug fix for composite bookmark-of
properties.create
/preview
: support large videos via async upload. We now pass media_category=tweet_video
to the chunked upload INIT
stage, and then make blocking STATUS
calls until the video is finished processing. (bridgy#1043)create
/preview
: allow bookmarks. (bridgy#1045)create
/preview
: allow non-Twitter replies, ie activities that include inReplyTo
URLs even if none of them point to a tweet. (bridgy#1063)get_activities
: support list ids as well as slugs.get_activities()
: raise ValueError
on invalid user_id
.scraped_to_activities()
, scraped_to_activity()
, and merge_scraped_reactions()
methods.summary
element (#157).Link
HTTP headers (eg rel=self
, rel=header
).get_comment()
: skip fetching comments from API if activity
kwarg is provided and contains the requested comment.#discussion_r...
fragments).aria-hidden="true"
to empty links (bridgy#947).&
, <
, and >
characters in bare mf2 content
properties (aaronpk/XRay#102).json_to_object()
: convert nickname
to username
.content_html
and content_text
are incorrectly lists instead of strings.limit
param for compatibility with Pleroma (bridgy#977).create()
: handle API errors and return the error message in the CreationResult
(bridgy#921).photo.displayName
so that it gets all the way into microformats2 JSON and HTML (#183).post_id()
.to_as1()
: for Create
activities, include the activity actor's data in the object's author (snarfed/bridgy-fed#75).to_as1()
: convert preferredUsername
to username
.from_as1()
: convert username
to preferredUsername
.from_as1()
: bug fix, make context
kwarg actually work.Breaking changes:
Non-breaking changes:
get_actor()
with user_id
.get_activites()
etc (#183).itunes:image
, itunes:author
, and itunes:category
.title
element (#177). Background.hfeed
correctly.article
or mention
tags in items with enclosures.hfeed
correctly.HEAD
support.input=html
. If a fragment is provided, only that specific element is extracted and converted. (#185)<code>
tags instead of converting them to `s so that GitHub renders HTML entities like >
inside them instead of leaving them escaped. Background.context
fields.html_to_activities()
: limit to h-entry
, h-event
, and h-cite
items (#192).cache
kwarg to Source.original_post_discovery()
now has no effect. webutil.util.follow_redirects()
has its own built in caching now.json
module to ujson
to speed up JSON parsing and encoding.duration
and size
support to ActivityStreams 1 and 2, RSS, and microformats2 HTML and JSON. microformats2 support is still emerging for both. Both integer seconds and ISO 8601 string durations are supported for duration
. Integer bytes is used for size
everywhere. microformats2 HTML also includes human-readable strings, eg 5.1 MB
. (#169)[preview]_create()
: detect attempts to upload images over 5MB and return an error.get_activities(scrape=True)
for scraping HTML from m.facebook.com. Requires c_user
and xs
cookies from a logged in session (snarfed/bridgy#886).<description>
element contents in CDATA
sections.<description>
with HTML <img>
tags (#175).from_activities()
bug fix: don't crash when converting multiple attachments to enclosures in a single item. (RSS only supports one enclosure per item, so we now only include the first, and log a warning if the activity has more.)Mention
tags to AS1 objectType
mention
(non-standard) and vice versa (snarfed/bridgy-fed#46).u-category
mf2.edge_media_to_parent_comment
field (#164).white-space: pre
CSS in HTML output.photo.php
as username in post URLs.white-space: pre
CSS back to converting newlines to <br>
s because some feed readers (eg NewsBlur) follow it too strictly and don't even line wrap.Breaking change: drop Google+ since it shuts down in March. Notably, this removes the googleplus
module.
<
/>
(snarfed/bridgy#850)./url
: Return HTTP 400 when fetching the user's URL results in an infinite redirect.Add delete()
. Currently includes Twitter and Flickr support.
create
/preview_create
bug fixes for issues and comments on private repos.delete()
and preview_delete()
for deleting tweets.delete()
and preview_delete()
for deleting photos.&
s in author URL and email address too. (Thanks sebsued!)Follow
support.create()
and preview_create()
: support RSVPs. Tweet them as normal tweets with the RSVP content. (snarfed/bridgy#818)create()
and preview_create()
: support alt text for images, via AS1 displayName
. (snarfed/bridgy#756).ignore_rate_limit
kwarg to get_activities()
.tag
support to create
/preview_create
to add label(s) to existing issues (snarfed/bridgy#811).<
, >
, and &
) in content in create()
and preview_create()
(snarfed/bridgy#810).get_activities()
and get_comment()
now return ValueError
instead of AssertionError
on malformed activity_id
and comment_id
args, respectively.get_activities()
bug fix for issues/PRs with no body text.alt
attribute in <img>
tags (snarfed/bridgy#756).get_activities()
with deleted issues and repos.object_to_json()
: convert tags to simple strings in the category
property, not full nested objects like h-card
s (#141)./issues
URL to be objectType issue
.This release is intentionally small and limited in scope to contain any impact of the Python 3 migration. It should be a noop for existing Python 2 users, and we've tested thoroughly, but I'm sure there are still bugs. Please file issues if you notice anything broken!
get_activities()
: Support @-prefixed usernames in user_id
.create()
and preview_create()
now only support RSVPs to individual instances of multi-instance events, to match the Facebook API itself._o.jpg
files instead of _s.jpg
.create()
bug fix for photo and image URLs with unicode characters.get_activities(user_id=...)
included the authenticated user's own recent photos, albums, and news publishes.author
) data from scraped profile pages.fetch_mf2
kwarg to json_to_object()
for fetching additional pages over HTTP to determine authorship.p-name
in HTML to prevent old flawed implied p-name handling (#131).share
verb handling in activity_to_json()
and activities_to_html()
(#134).<br>
s (#130).updated
and published
.p-name
logic.)get_activities()
etc.u-photo
, u-video
, and u-audio
classes more often and consistently.atom_to_activities()
for converting full feed documents.rel=alternate
link as well as actor/author URL (#151).as2
module includes to_as1()
and from_as1()
functions. Currently supported: articles, notes, replies, likes, reposts, events, RSVPs, tags, attachments.atom_to_activity()
function for converting Atom to AS1.as2
value for format
and input
. Revise existing ActivityStreams and microformats2 value names to as1
, as1-xml
, and mf2-json
. Old values activitystreams
, json
, json-mf2
, and xml
are still accepted, but deprecated.get_blocklist()
.post_id()
now validates ids more strictly before returning them.u-featured
to ActivityStreams image
.h-event
support.) when rendering locations as HTML.
post_id()
now validates ids more strictly before returning them.activity_to_atom()
function that renders a single top-level <entry>
instead of <feed>
.reader
query param for toggling rendering decisions that are specific to feed readers. Right now, just affects location: it's rendered in the content when reader=true
(the default), omitted when reader=false
.activity:object-type
and activity:verb
elements when they have values.thr:in-reply-to
from object.inReplyTo
as well as activity.context.inReplyTo
.preview_create()
.get_activities()
is passed group_id='@search'
but not search_query
.--enable-unicode=ucs2
, which is the default on Mac OS X, Windows, and older *nix.news.publish
actions.xml:base
.u-like
and u-repost
properties.