Amazon RDS, GCP Cloud SQL, Azure, Heroku support?

[jqnatividad]

RDS comes with pre-packaged extensions... are there any talks with AWS to include timescale in their supported extensions, given that timescaledb uses the Apache 2.0 license?

[stalltron]

Hi Joel, thanks for this question. We're connected with the folks at RDS but it's most helpful if they receive outside requests: https://aws.amazon.com/rds/postgresql/faqs/

Q: What are the extensions supported by RDS for PostgreSQL? You can check the list of supported extensions in the Amazon RDS User Guide. To request support for additional extensions, you can send an email to rds-postgres-extensions-request@amazon.com with the extension name and use case.

We would also recommend that you ask your AWS Account Manager for the TimescaleDB extension to be made available on RDS.

Would then be useful if you could let us know (privately) their feedback and we can reassess how to expedite the process of becoming an extension. Hope this helps.

[UPDATE] We've received many other requests to be supported on GCP Cloud SQL, Heroku, and Azure, so we wanted to list the appropriate places to submit extension requests for those platforms here, along with RDS, and keep the information in one place.

• GCP Cloud SQL: https://issuetracker.google.com/issues/64602105
• Heroku: https://help.heroku.com/tickets/new?q=extension%20for%20timescaledb (no ticket yet)
• Azure: https://feedback.azure.com/forums/597976-azure-database-for-postgresql/suggestions/20055946-timescaledb-extension

[orlando]

@mfreed do you know if there's an update about this?

[stalltron]

Hi @orlando we haven't announced a timeline and some of it is out of our control. We'll provide updates as we have them.

[orlando]

Thanks @stalltron, having that will be a huge win for timescaledb. Last week I tried it with metabase and works OTB (expected since is still postgres) but it was really good to have graphs from a time series database in less than 10 minutes

[misham]

Is it possible to run TimescaleDB as a "frontend" on a separate machine that connects to an RDS instance?

[stalltron]

@misham Interesting question.. We can be run on EC2, so the question then becomes can hosted offerings like RDS be run alongside self-managed EC2 instances? I'm not entirely sure, but will look into this...

[misham]

If you're looking into using RDS as a slave and PG running as master on an EC2 instance, that doesn't work - at least I haven't been able to figure out how to make that work.

I was wondering if it's possible to make TimescaleDB run as a separate service that uses PG as a "backend" and then have the app talk only to TimescaleDB.

[stalltron]

@misham Ah, ok. Can you share more about the use case you have in mind? I'm not sure I understand "uses PG as a backend," since data stored in TimescaleDB is effectively stored in PostgreSQL. Please email me (andrew@timescale.com) if there's stuff you can't share publicly.

[ghost]

This is exactly the use case I have.

So the idea is you have a clustered TimescaleDB service living in AWS EC2 (presumably an autoscaling group), that's load balanced (I'm new to this tool and I'm just assuming this can be done) and that that clustered service is configured to point to an RDS endpoint for storage.

I'm a little surprised that this isn't something timescaleDB can't do today out of box. It seemed obvious to me that such a tool would be able to consume arbitrarily defined PostgreSQL backends for storage; is this not the case?

I've only just discovered this tool, so forgive my ignorance, but my initial assumption was that I'd see some kind of config file where the default postgresql location is set to localhost, but can be configured as necessary. If this can't be done today then it strikes me as a necessary, and very basic, feature.

This would solve the problem of people seeking support for database services on every possible hosting platform without needing the input of those hosting platforms, and it's a pretty commonly expected behavior from any tooling that writes to/reads from a database (that is, the ability to configure the database endpoint/connection-string/credentials/terminology-of-choice).

[orlando]

@TalosThoren TimescaleDB is not a service that sits in front of a database, is a PostgreSQL extension (ex. PostGIS) you install to add new capabilities to it (http://docs.timescale.com/latest/using-timescaledb/hypertables). You can do it without problems since you are using EC2, but with managed services like RDS is not possible yet (AWS needs to whitelist the extension first).

Or maybe I'm not getting what you are trying to say

[ghost]

@orlando, that makes perfect sense and explains the limitations preventing this from being deployed in my expected use case. Thank you for clarifying.

[akulkarni]

Thank you @orlando, that is correct.

@TalosThoren - If you'd like to expedite the AWS whitelisting process, please feel free to follow the instructions at the top of this issue.

Thanks!

[lambdaq]

Might as well add Aliyun to that list.

They wrote multiple blogs but not supported in official pg extensions.

https://yq.aliyun.com/articles/241194

https://yq.aliyun.com/articles/73537

[lambdaq]

Update: I contacted Aliyun (AlibabaCloud) support via ticket system (https://workorder.console.aliyun.com), their response is

以上您建议我们会评估分析一下关于功能需求，但是目前还没有得到评估计划、相关功能是否可以支持及确认支持时间。

Translation: We will evaluate this, but we dont have a plan or confirmation yet.

List of installed plugins are:

psql (9.2.23, server 9.4.10)
WARNING: psql version 9.2, server version 9.4.
         Some psql features might not work.
Type "help" for help.

db=> \dx
                 List of installed extensions
  Name   | Version |   Schema   |         Description
---------+---------+------------+------------------------------
 plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
(1 row)

db=>  SELECT * FROM pg_available_extensions;
             name             | default_version | installed_version |                                                       comment

------------------------------+-----------------+-------------------+---------------------------------------------------------------------------------------------------------------------
 address_standardizer         | 2.2.2           |                   | Used to parse an address into constituent elements. Generally used to support geocoding address normalization step.
 chkpass                      | 1.0             |                   | data type for auto-encrypted passwords
 test_shm_mq                  | 1.0             |                   | Test code for shared memory message queues
 postgis_topology             | 2.2.2           |                   | PostGIS topology spatial types and functions
 pgrowlocks                   | 1.1             |                   | show row-level locking information
 jsonbx                       | 1.0             |                   | Jsonb extension
 pg_hint_plan                 | 1.1.3           |                   |
 dblink                       | 1.1             |                   | connect to other PostgreSQL databases from within a database
 adminpack                    | 1.0             |                   | administrative functions for PostgreSQL
 earthdistance                | 1.0             |                   | calculate great-circle distances on the surface of the Earth
 plperlu                      | 1.0             |                   | PL/PerlU untrusted procedural language
 plcoffee                     | 1.4.2           |                   | PL/CoffeeScript (v8) trusted procedural language
 tcn                          | 1.0             |                   | Triggered change notifications
 pg_freespacemap              | 1.0             |                   | examine the free space map (FSM)
 timetravel                   | 1.0             |                   | functions for implementing time travel
 oss_fdw                      | 1.1             |                   | foreign-data wrapper for flat file access
 plls                         | 1.4.2           |                   | PL/LiveScript (v8) trusted procedural language
 imgsmlr                      | 1.0             |                   | image similarity module
 intarray                     | 1.0             |                   | functions, operators, and index support for 1-D arrays of integers
 pg_sphere                    | 1.0             |                   | spherical objects with useful functions, operators and index support
 varbitx                      | 1.0             |                   | varbit functions pack
 pgrouting                    | 2.0.0           |                   | pgRouting Extension
 pageinspect                  | 1.2             |                   | inspect the contents of database pages at a low level
 pgstattuple                  | 1.2             |                   | show tuple-level statistics
 intagg                       | 1.0             |                   | integer aggregator and enumerator (obsolete)
 citext                       | 1.0             |                   | data type for case-insensitive character strings
 ali_decoding                 | 0.0.1           |                   | ali_decoding
 insert_username              | 1.0             |                   | functions for tracking who changed a table
 xml2                         | 1.0             |                   | XPath querying and XSLT
 postgis                      | 2.2.2           |                   | PostGIS geometry, geography, and raster spatial types and functions
 pgcrypto                     | 1.1             |                   | cryptographic functions
 pltclu                       | 1.0             |                   | PL/TclU untrusted procedural language
 pg_stat_statements           | 1.2             |                   | track execution statistics of all SQL statements executed
 orafce                       | 3.6             |                   | Functions and operators that emulate a subset of functions and packages from the Oracle RDBMS
 hstore                       | 1.3             |                   | data type for storing sets of (key, value) pairs
 lo                           | 1.0             |                   | Large Object maintenance
 plperl                       | 1.0             |                   | PL/Perl procedural language
 q3c                          | 1.5.0           |                   | q3c sky indexing plugin
 postgres_fdw                 | 1.0             |                   | foreign-data wrapper for remote PostgreSQL servers
 zhparser                     | 1.0             |                   | a parser for full-text search of Chinese
 tablefunc                    | 1.0             |                   | functions that manipulate whole tables, including crosstab
 refint                       | 1.0             |                   | functions for implementing referential integrity (obsolete)
 pg_prewarm                   | 1.0             |                   | prewarm relation data
 btree_gin                    | 1.0             |                   | support for indexing common datatypes in GIN
 autoinc                      | 1.0             |                   | functions for autoincrementing fields
 postgis_tiger_geocoder       | 2.2.2           |                   | PostGIS tiger geocoder and reverse geocoder
 test_parser                  | 1.0             |                   | example of a custom parser for full-text search
 pg_buffercache               | 1.0             |                   | examine the shared buffer cache
 btree_gist                   | 1.0             |                   | support for indexing common datatypes in GiST
 fuzzystrmatch                | 1.0             |                   | determine similarities and distance between strings
 moddatetime                  | 1.0             |                   | functions for tracking last modification time
 address_standardizer_data_us | 2.2.2           |                   | Address Standardizer US dataset example
 tsearch2                     | 1.0             |                   | compatibility package for pre-8.3 text search functions
 worker_spi                   | 1.0             |                   | Sample background worker
 pg_awr                       | 1.0             |                   | PostgreSQL awr
 pltcl                        | 1.0             |                   | PL/Tcl procedural language
 file_fdw                     | 1.0             |                   | foreign-data wrapper for flat file access
 smlar                        | 1.0             |                   | compute similary of any one-dimensional arrays
 www_fdw                      | 0.1.8           |                   | WWW FDW - extension for handling different web services
 rdkit                        | 3.4             |                   | Cheminformatics functionality for PostgreSQL.
 plpythonu                    | 1.0             |                   | PL/PythonU untrusted procedural language
 ltree                        | 1.0             |                   | data type for hierarchical tree-like structures
 dict_int                     | 1.0             |                   | text search dictionary template for integers
 unaccent                     | 1.0             |                   | text search dictionary that removes accents
 uuid-ossp                    | 1.0             |                   | generate universally unique identifiers (UUIDs)
 pg_trgm                      | 1.1             |                   | text similarity measurement and index searching based on trigrams
 plpgsql                      | 1.0             | 1.0               | PL/pgSQL procedural language
 plpython2u                   | 1.0             |                   | PL/Python2U untrusted procedural language
 seg                          | 1.0             |                   | data type for representing line segments or floating-point inte rvals
 plv8                         | 1.4.2           |                   | PL/JavaScript (v8) trusted procedural language
 dict_xsyn                    | 1.0             |                   | text search dictionary template for extended synonym processing
 cube                         | 1.0             |                   | data type for multidimensional cubes
 sslinfo                      | 1.0             |                   | information about SSL certificates
 isn                          | 1.0             |                   | data types for international product numbering standards
(74 rows)

[lambdaq]

@orlando

TimescaleDB is not a service that sits in front of a database, is a PostgreSQL extension (ex. PostGIS) you install to add new capabilities to it (http://docs.timescale.com/latest/using-timescaledb/hypertables).

Let's think about this, it's ok to setup a pg isntance on ec2, however, the main problem is the extremely low IOPS.

What if we can setup a pg broker with timescale extension on ec2, but we don't do data persistency on broker, instead we use hosted RDS to do the heavy-lifting for actual read/writes.

The pg broker instance is only for query planing.

Is this doable?

[mfreed]

@lambdaq Thanks for reaching out to Aliyun and providing that data.

I should also add that TimescaleDB currently supports Postgres version 9.6 and 10. It looks like at least the version of the Aliyun PG instance you are using is 9.4 (which was released in late 2014), but we don't currently have any plans to backport to 9.4.

[mfreed]

@lambdaq Regarding your postgres broker idea, is there a particular software codebase you are referring to? Timescale can be utilized with remote storage via FDW, but also does run "inside" a Postgres database instance itself.

That said, I'm not sure I follow/agree with the assumption behind your question, which is that PG databases deployed on EC2 suffer from low IOPS. This seems mostly due to the type of storage selected with EC2: local SSD, remote SSD, remote HDD, provisioned vs. unprovisioned IOPS, etc.

We've certainly had no problems reaching steady-state traffic rates of 100K+ rows / second on these cloud VMs with the modest provisioning of network-attached SSD.

[rymoore]

I've contacted AWS RDS multiple times via email to request TimeScale support... no response

[latortuga]

I've sent requests to Heroku and AWS RDS, please if this is relevant to you, reach out! It takes 5 minutes!

[selbyk]

@latortuga Best way to send feature requests to AWS?

[latortuga]

There are lots of links up this issue thread.

On Fri, Apr 27, 2018, 3:21 PM Selby Kendrick notifications@github.com wrote:

@latortuga https://github.com/latortuga Best way to send feature requests to AWS?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/timescale/timescaledb/issues/65#issuecomment-385083806, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAko40th6QBlnBb37Qk8g6D5fiXqWkEks5ts33ggaJpZM4Nn5rR .

[rawberg]

It looks like Aiven recently added TimescaleDB support to their cloud managed Postgres option. Not affiliated and haven't tried them yet.

[craiggenner]

I've raised a ticket with AWS support to request this, and I'll also email the Aurora Postgresql Product manager to follow this up and see what he says.

[craiggenner]

I've had this response to my ticket:

"To request support for additional extensions/plugins, you can send an email to rds-postgres-extensions-request@amazon.com with the extension/plugin name and use case. "

My request has been put to the team though (along with the rest)

[Helmi]

sent an email to that AWS account too without a response but that seems to be the way that works. Wonder how this topic hasn't come forward after over a year of "contact" with the AWS RDS team @stalltron @mfreed - are they declining to bring TimescaleDB on or is there just work to be done? Can you give any status update or outlook? Any chance to see timescaleDB on RDS anytime soon?

[craiggenner]

I met with some members of the RDS team and it would appear its more likely they have so many core features they want to implement first.

But yes don't expect a reply to the email address.

[jwg2s]

I've also emailed rds-postgres-extensions-request@amazon.com and will bring up with our AWS account rep - would love to see this get supported!

[ashwinpromethean]

Yes, would like to see this as well. We are sticking to Postgres as of now for over a year (and come back to this page once a month), primarily because timescale isnt supported by RDS.

Hoping for the best.

[mmouterde]

I met an AWS Account Manager yesterday and he adds us as customer reference to this feature requirement. No garanties but that should contribute.

[maksimluzik]

Please go add a comment to the AWS forums so they would speed up the integration here: https://forums.aws.amazon.com/thread.jspa?messageID=799694&#799694

[austinh]

Any update on this? Would love to help in anyway I can to speed this along. I’ve created tickets at AWS and Heroku for this but no response

[sachaarbonel]

Digital Ocean has released Managed Databases. We might have a chance to see a positive response from them.

[pjebs]

This is probably the reason why AWS isn't supporting this postgres extension for RDS: Amazon Timestream: https://aws.amazon.com/timestream/?hp=r

[alanhamlett]

This is probably the reason why AWS isn't supporting this postgres extension for RDS: Amazon Timestream

AWS has other competing products so I'm still hoping to see Timescale supported in RDS PG extensions.

[Tyler-Murphy]

What does the new license mean for these integrations?

[mehiel]

AFAICT it simply means that they should pay TimescaleDB to provide their customers the extension in an "as a service" manner.

That of course is a reasonable decision on TimescaleDB's side as they want to profit as well on that case BUT it will make integrations harder to happen as it'll cost all those service providers more and they'll have to pass the cost to the end customer at the end. On top of that, AWS especially is rumored to not actually play well with others.

On the other hand there is a lot of discussion in the OSS community for that kind of licensing. Mongo's decision and actual license seems like it backfired in the end (?) (see https://www.theregister.co.uk/2019/01/17/red_hat_mongodb/)

It's interesting to see what's going to happen. It's important for all these data related businesses to find a way to profit because the won't survive otherwise (see Riak) and that's a problem for the community that wants to avoid vendor lock-in and trust some stable tech that will not extinguish in the next couple of years.

[mfreed]

To be clear, the various cloud providers can certainly deploy the Apache 2 version of TimescaleDB; all capabilities to date have been licensed and remain available under the Apache 2.

That has been one confusion about the announcement of the Timescale License. Unlike Mongo and Confluent, we did not relicense code from the Apache 2 to TSL. We simply "preannounced" that some future capabilities will be licensed under something different than the Apache 2.

What this does mean is that these future capabilities that are TSL licensed can't be deployed by the DBaaS / cloud providers, without some additional commercial relationship with Timescale. The cloud providers will continue to be able to deploy the OSS version of TimescaleDB.

[mehiel]

Thanks for the clarifications @mfreed.

TimescaleDB's decision seems way more balanced and I think they'll be welcomed by both the community and the cloud providers. Time will tell. Keep it up!

[chickahoona]

First of all I understand that people behind timescale want to make a living and the decision behind TSL and so on, yet afaik that is a deathshot for all dreams of getting Timescale on AWS / GCP / ... No cloud provider will install it when by any chance a future version might

a) have an addon or whatever bundled with this TSL. It would open the doors for a potential lawsuit and is just not worth the trouble.

b) tomorrow you would decide that only your TSL version would support Postgres 10, then that would mean that they would not be able to upgrade to Postgres 10 or would have to get a shitstorm from all their customers. (this also applies to all other potential restrictions that you might add to a new timescaledb version)

You can argue whatever you like about "It wont happe" or "they can still use the apache 2.0" ... It just does not matter to them. Also to have this discussion here now does not really matter anymore. The guys behind timescaledb for sure knew all of that when they made the decision and made it anyway, fully accepting this.

So if you want it, you have to host it yourself.

[windbender]

Yo AWS RDS people, Azure is going to do it. How about you ?

AZURE: " Unplanned → Planned

We’ve been working with TimescaleDB team to enable it in Azure Database for PostgreSQL service. We expect TimescaleDB extension to be available to customers of our service worldwide in 6-8 weeks from now.

Nik on behalf of the Azure Database for PostgreSQL team

  | Azure Database for PostgreSQL Product TeamAdmin, Microsoft Azure "

[mehiel]

Citus acquisition then this ^. There is something happening for sure with MS :) Good news!

[pjebs]

I'll believe it when I see it. It's a shame they won't implement it for a non-Aurora postgres RDS. Aurora is too expensive for my use case.

[austinh]

They have timescaledb on aurora RDS?

[pjebs]

@austinh It's supposedly a few weeks away. But I'll believe it when I see it.

[chanon]

I would really love TimescaleDB to be available on Digital Ocean.

[JeroneGagliano]

Any update on this? Would love to help in anyway I can to speed this along. I’ve created tickets at AWS and Heroku for this but no response

I submitted a ticket request for timescale add-on to Heroku as well.

[pjebs]

I think Digital Ocean will be more receptive from a business perspective. They just launched their managed service and it's a great opportunity to bring in new customers Immediately:https://blog.digitalocean.com/announcing-managed-databases-for-postgresql/

@akulkarni

[consense]

@pjebs no need to be hopeful as it is already confirmed that the DigitalOcean Managed Database offering contains Timescale as plugin option by default in the status quo as of today :-)

https://gist.github.com/peterc/e4f7a288ed0eb7e4ffe2d8383a086306

[ehamberg]

Live from DigitalOcean's ~(limited availability)~ PostgreSQL offering:

[edencorbin]

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.Extensions.11x

Still not supported at AWS, anybody know when it might be?

I also don't see it over at Azure: https://docs.microsoft.com/en-us/azure/postgresql/concepts-extensions

Anything new on MS support?