what are improper lists?

While looking at the code of the all/1 function in the Ecto.Adapters.Postgres.Connection module (see https://github.com/dwyl/alog/issues/38 for the "why") we can see the returned value is

[select, from, join, where, group_by, having, window, combinations, order_by, limit, offset | lock]

Let's use the syntax [1, 2, 3] and [1, 2 | 3] to understand the difference between lists and improper lists. We know that [1, 2, 3] in Elixir represents a list. The cons operator | allow us to write the list as [1 | [2 | [3 | [] ] ] ]. This means that a list is the "combination" of a head and a tail:

head: 1, tail [2, 3]
head 2, tail [3]
head 3, tail [] (empty list)

A "normal" list in Elixir has always contains an empty list tail

If we now apply the cons operator to the list [1, 2 | 3](head 1 and tail [2 | 3]) we can see that we can't get an empty list for tail and the last tail is [2 | 3]. This is called an improper list.

An improper list is a list which doesn't contain an empty list for the last tail

[ ] Add a new section under data-structures to explain improper list

From the List module documenation (https://hexdocs.pm/elixir/master/List.html):

Although improper lists are generally avoided, they are used in some special circumstances like iodata and chardata entities (see the IO module).

We can see that in the case of the Postgres adapter, the improper list returned in the all function is then used with the function IO.iodata_binary/1 here https://github.com/elixir-ecto/ecto_sql/blob/3a72eb111f8613c8eada296bcd3dd89883a52b5a/lib/ecto/adapters/sql.ex#L134

IO.iodata_to_binary(@conn.all(query))}}

where @conn is the Postgres adapter module.

The documentation of iodata_to_binary says:

Converts iodata (a list of integers representing bytes, lists and binaries) into a binary. The operation is Unicode unsafe.

an iodata in Elixir is defined as:

see:

Because String in Elixir are binaries and improper list are still a list so we can say that an improper list of string is an iodata. For example [" from ", " table ", " as " | " table name "] is an iodata

and if we now run the iodata_to_binary function we get the string concatenation as a result:

iex> IO.iodata_to_binary([" from ", " table ", " as " | " table name "])
iex > " from  table  as  table name "

So at the end my understanding of using improper lists with iodata_to_binary is a fancy way (ie enhance performance) to do a binaries/string concatenation.

My goal is now to read in more detail http://www.evanmiller.org/elixir-ram-and-the-template-of-doom.html to understand how this performance enhancement is done. I think that's one of the reason Phoenix template are fast to process

dwyl / learn-elixir

what are improper lists? #120