AG codes and decoders

[kwankyu]

We add AG codes and unique decoders for AG codes.

In Attachments below, there are Jupyter notebooks demonstrating the new features provided by this ticket. These were presented in 2019 SIAM Conference on Algebraic Geometry.

---

The author was supported by NRF of Korea 2018, 2019, 2020

Component: coding theory

Author: Kwankyu Lee

Branch/Commit: 7097dc7

Reviewer: Travis Scrimshaw

Issue created by migration from https://trac.sagemath.org/ticket/27957

[kwankyu]

Author: Kwankyu Lee

[kwankyu]

Branch: u/klee/27957

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Commit: 1295929

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

1295929

Add AG codes and decoders

[kwankyu]

Dependencies: #27873

[kwankyu]

Changed dependencies from #27873 to #27873, #27634

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

965f050	Merge branch 'curves-2-trac27954' into curves-3-trac27955
42ad3b9	Merge branch 'curves-3-trac27955' into curves-4-trac27956
9fa25f4	Merge branch 'curves-4-trac27956' into ag-codes-base
04d0cb9	fix
94ca618	fix
25139ce	pyflakes fix
4fb3475	Merge branch 'curves-2-trac27954' into curves-3-trac27955
f102381	Merge branch 'curves-3-trac27955' into curves-4-trac27956
92cefd2	Merge branch 'curves-4-trac27956' into ag-codes-base
6285d99	Add AG codes and decoders

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 1295929 to 6285d99

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

3dc2b79

Add AG codes and decoders

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 6285d99 to 3dc2b79

[embray]

comment:10

Tickets still needing working or clarification should be moved to the next release milestone at the soonest (please feel free to revert if you think the ticket is close to being resolved).

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 3dc2b79 to 2057fcd

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

2057fcd

Add AG codes and decoders

[kwankyu]

Description changed:

--- 
+++ 
@@ -1,6 +1,4 @@
-Evaluation and differential AG codes are added. 
-
-Unique decoders for AG codes are addded.
+AG codes and unique decoders for AG codes are added.

 ---
 The author was supported by NRF of Korea 2018, 2019

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 2057fcd to bd43bff

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

bd43bff

Add AG codes and decoders

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from bd43bff to f00dca6

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

f00dca6

Add AG codes and decoders

[kwankyu]

Description changed:

--- 
+++ 
@@ -1,4 +1,8 @@
 AG codes and unique decoders for AG codes are added.
+
+This ticket will be a meta-ticket and will have child tickets to merge the code in several steps. This process will begin after the dependencies including #27873, #27634 are merged to Sage first.
+
+In the meantime, the branch attached to the present ticket will provide the stable code tested with the latest release of Sage.

 ---
 The author was supported by NRF of Korea 2018, 2019

[kwankyu]

Description changed:

--- 
+++ 
@@ -1,8 +1,8 @@
 AG codes and unique decoders for AG codes are added.

-This ticket will be a meta-ticket and will have child tickets to merge the code in several steps. This process will begin after the dependencies including #27873, #27634 are merged to Sage first.
+This ticket will be a meta-ticket and will have child tickets to merge the code in several steps. This process will begin after the dependencies are merged to Sage first.

-In the meantime, the branch attached to the present ticket will provide the stable code tested with the latest release of Sage.
+In the mean while, the branch attached to the present ticket will provide the stable code tested with the latest release of Sage.

 ---
 The author was supported by NRF of Korea 2018, 2019

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from f00dca6 to 9978f4d

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

9978f4d

Add AG codes and decoders

[embray]

comment:20

Moving tickets from the Sage 8.8 milestone that have been actively worked on in the last six months to the next release milestone (optimistically).

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 9978f4d to 1154e85

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

1154e85

Add AG codes and decoders

[kwankyu]

Description changed:

--- 
+++ 
@@ -2,7 +2,7 @@

 This ticket will be a meta-ticket and will have child tickets to merge the code in several steps. This process will begin after the dependencies are merged to Sage first.

-In the mean while, the branch attached to the present ticket will provide the stable code tested with the latest release of Sage.
+In the mean while, the branch attached to the present ticket will provide the stable code tested with the latest Sage release, version 8.8.

 ---
 The author was supported by NRF of Korea 2018, 2019

[kwankyu]

Codes on affine line

[kwankyu]

Attachment: Reed Solomon codes.ipynb.gz

Codes on projective line

[kwankyu]

Attachment: Goppa codes.ipynb.gz

Codes on affine plane curve

[kwankyu]

Attachment: Codes on Hermitian curve.ipynb.gz

Attachment: Codes on Klein quartic.ipynb.gz

Codes on projective plane curve

[kwankyu]

Codes on space curve

[kwankyu]

Attachment: Codes on GK curves I.ipynb.gz

Codes on space curve

[kwankyu]

Description changed:

--- 
+++ 
@@ -2,7 +2,9 @@

 This ticket will be a meta-ticket and will have child tickets to merge the code in several steps. This process will begin after the dependencies are merged to Sage first.

-In the mean while, the branch attached to the present ticket will provide the stable code tested with the latest Sage release, version 8.8.
+In the mean while, the branch attached to the present ticket will provide the stable code tested with the latest Sage release, version 8.8. 
+
+In Attachments below, there are Jupyter notebooks demonstrating the new features provided by this ticket. These were presented in 2019 SIAM Conference on Algebraic Geometry.

 ---
 The author was supported by NRF of Korea 2018, 2019

[kwankyu]

comment:23

Attachment: Codes on GK curves II.ipynb.gz

[kwankyu]

Changed dependencies from #27873, #27634 to #27873

[kwankyu]

Changed dependencies from #27873 to none

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 1154e85 to 27e32bd

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

27e32bd

Add AG codes and decoders

[kwankyu]

Description changed:

--- 
+++ 
@@ -2,9 +2,9 @@

 This ticket will be a meta-ticket and will have child tickets to merge the code in several steps. This process will begin after the dependencies are merged to Sage first.

-In the mean while, the branch attached to the present ticket will provide the stable code tested with the latest Sage release, version 8.8. 
+In the mean while, the branch attached to the present ticket will provide the stable code tested with the latest Sage release. 

 In Attachments below, there are Jupyter notebooks demonstrating the new features provided by this ticket. These were presented in 2019 SIAM Conference on Algebraic Geometry.

 ---
-The author was supported by NRF of Korea 2018, 2019
+The author was supported by NRF of Korea 2018, 2019, 2020

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 27e32bd to e03bf5e

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e03bf5e

Add AG codes and decoders

[kwankyu]

Description changed:

--- 
+++ 
@@ -1,8 +1,4 @@
-AG codes and unique decoders for AG codes are added.
-
-This ticket will be a meta-ticket and will have child tickets to merge the code in several steps. This process will begin after the dependencies are merged to Sage first.
-
-In the mean while, the branch attached to the present ticket will provide the stable code tested with the latest Sage release. 
+We add AG codes and unique decoders for AG codes.

 In Attachments below, there are Jupyter notebooks demonstrating the new features provided by this ticket. These were presented in 2019 SIAM Conference on Algebraic Geometry.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

c5baba9	Merge branch 'develop' into ag-codes-trac27957
eeba013	Fix a bug in constant field extension of AG code"

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from e03bf5e to eeba013

[tscrim]

comment:32

Looks good overall. I just have a few comments.

I think it is better to be more specific about where you are importing stuff

-from sage.modules.all import vector
-from sage.matrix.all import matrix, MatrixSpace
+from sage.modules.free_module_element import vector
+from sage.matrix.constructor import matrix
+from sage.matrix.matrix_space import MatrixSpace

This can help limit circular imports.

I don't understand why you have the function cartier_fixed in CartierCode.__init__. It is only called once and just has one return statement at the bottom.

I don't think it is a good idea to assume the doctests will be run in order. (In fact, I think there is a ticket looking to run them in any order.) I wouldn't rely on a saved object. I would instead suggest having a pickle in the file (or a separate test file) as a string (or a pickle stored somewhere) that you can load.

-As well known, the two kinds of AG codes are dual to each other. ::
+As is well known, the two kinds of AG codes are dual to each other. ::

Something else you can do for __eq__ is first test if self is other: to short-circuit when comparing against the same object.

In DifferentialAGCodeUniqueDecoder._encode and _decode, you have TESTS: with one colon.

Do you want _Decoder_K to be hidden from the documentation? Since it is a key component, it might be good to include it.

Since _Decoder_K and _Decoder_K_extension and their subclasses seem like the key workhorses, would it make sense to include them as a separate Cython file and make them cdef classes?

I am not sure about variable scope here:

        # compute xR
        for xR in Q.divisor(gamma).basis_function_space():
            if xR.valuation(Q) == -gamma:
                break

I think it would be better to do

        def compute_xR():
            for xR in Q.divisor(gamma).basis_function_space():
                if xR.valuation(Q) == -gamma:
                    return xR
        xR = compute_xR()

I thought Python3 didn't allow variables to leak out of scope like that anymore? Well, I find this to be a little safer. Another option would be

        for b in Q.divisor(gamma).basis_function_space():
            if b.valuation(Q) == -gamma:
                xR = b
                break

[kwankyu]

comment:33

Replying to @tscrim:

Looks good overall. I just have a few comments.

I appreciate your comments. Thank you.

I think it is better to be more specific about where you are importing stuff

-from sage.modules.all import vector
-from sage.matrix.all import matrix, MatrixSpace
+from sage.modules.free_module_element import vector
+from sage.matrix.constructor import matrix
+from sage.matrix.matrix_space import MatrixSpace

This can help limit circular imports.

Okay.

I don't understand why you have the function cartier_fixed in CartierCode.__init__. It is only called once and just has one return statement at the bottom.

To encapsulate the code in cartier_fixed. I didn't want the variables defined inside to affect the code outside. Moreover the code is natural for a general divisor E, but is applied to the specific divisor G-D.

I don't think it is a good idea to assume the doctests will be run in order. (In fact, I think there is a ticket looking to run them in any order.) I wouldn't rely on a saved object. I would instead suggest having a pickle in the file (or a separate test file) as a string (or a pickle stored somewhere) that you can load.

But then the pickle is not updated, and hence the tests would be for old objects. Isn't it against the purpose of the testing?

-As well known, the two kinds of AG codes are dual to each other. ::
+As is well known, the two kinds of AG codes are dual to each other. ::

Something else you can do for __eq__ is first test if self is other: to short-circuit when comparing against the same object.

In DifferentialAGCodeUniqueDecoder._encode and _decode, you have TESTS: with one colon.

Okay.

Do you want _Decoder_K to be hidden from the documentation? Since it is a key component, it might be good to include it.

It is a key component. But the user does not need to interact with it. So the class name starts with an underline. It works in the background. The classes before it define the interface for the user. So I think it should be hidden.

Since _Decoder_K and _Decoder_K_extension and their subclasses seem like the key workhorses, would it make sense to include them as a separate Cython file and make them cdef classes?

Right they are the key workhorses. Maybe a good idea. Though I don't have much experiences with Cython files, I will try.

I am not sure about variable scope here:

        # compute xR
        for xR in Q.divisor(gamma).basis_function_space():
            if xR.valuation(Q) == -gamma:
                break

I think it would be better to do

        def compute_xR():
            for xR in Q.divisor(gamma).basis_function_space():
                if xR.valuation(Q) == -gamma:
                    return xR
        xR = compute_xR()

I thought Python3 didn't allow variables to leak out of scope like that anymore? Well, I find this to be a little safer. Another option would be

        for b in Q.divisor(gamma).basis_function_space():
            if b.valuation(Q) == -gamma:
                xR = b
                break

I will use the last version. Thanks. By the way, the second version is in the same vein with the function cartier_fixed.

[tscrim]

comment:34

Replying to @kwankyu:

Replying to @tscrim:

I don't understand why you have the function cartier_fixed in CartierCode.__init__. It is only called once and just has one return statement at the bottom.

To encapsulate the code in cartier_fixed. I didn't want the variables defined inside to affect the code outside. Moreover the code is natural for a general divisor E, but is applied to the specific divisor G-D.

If it is natural, then I would suggest separating it out as a separate function. As long as the variables you use have distinct names, there should be no difference inside a function than outside.

I don't think it is a good idea to assume the doctests will be run in order. (In fact, I think there is a ticket looking to run them in any order.) I wouldn't rely on a saved object. I would instead suggest having a pickle in the file (or a separate test file) as a string (or a pickle stored somewhere) that you can load.

But then the pickle is not updated, and hence the tests would be for old objects. Isn't it against the purpose of the testing?

It would use the current object (assuming it isn't pickled in some non-standard way) when it is reconstructed. I would have to double-check with the pickling protocols, but the pickle data should only need to be updated in limited circumstances.

Do you want _Decoder_K to be hidden from the documentation? Since it is a key component, it might be good to include it.

It is a key component. But the user does not need to interact with it. So the class name starts with an underline. It works in the background. The classes before it define the interface for the user. So I think it should be hidden.

I don't see why it needs to be hidden in the documentation. The documentation is also there for the maintainer as well as the user. Although if this is in a separate file, then you could have the "main" class documentation isolated from this.

Since _Decoder_K and _Decoder_K_extension and their subclasses seem like the key workhorses, would it make sense to include them as a separate Cython file and make them cdef classes?

Right they are the key workhorses. Maybe a good idea. Though I don't have much experiences with Cython files, I will try.

To Cythonize it, you just need to move your code into a .pyx. To make it a bit better, you add cdef before each class and create .pxd file (the .pxd is like a .h header file in C) with those class declarations. There should be a number of easily accessible examples for this; e.g., groups/group.pxd.

I am not sure about variable scope here:

I will use the last version. Thanks. By the way, the second version is in the same vein with the function cartier_fixed.

The key difference is the return statement aborting the run part of the way through the loop that should set a variable.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from eeba013 to c7df4e6

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

241302d	Merge branch 'develop' into ag-codes-trac27957
c7df4e6	Fixes for reviewer comments

[kwankyu]

comment:36

Replying to @tscrim:

To encapsulate the code in cartier_fixed. I didn't want the variables defined inside to affect the code outside. Moreover the code is natural for a general divisor E, but is applied to the specific divisor G-D.

If it is natural, then I would suggest separating it out as a separate function. As long as the variables you use have distinct names, there should be no difference inside a function than outside.

I inlined the code removing cartier_fixed encapsulation. I am okay with this, but it seems to me that the code lost a little bit of readibility.

But then the pickle is not updated, and hence the tests would be for old objects. Isn't it against the purpose of the testing?

It would use the current object (assuming it isn't pickled in some non-standard way) when it is reconstructed. I would have to double-check with the pickling protocols, but the pickle data should only need to be updated in limited circumstances.

Okay. Done.

It is a key component. But the user does not need to interact with it. So the class name starts with an underline. It works in the background. The classes before it define the interface for the user. So I think it should be hidden.

I don't see why it needs to be hidden in the documentation. The documentation is also there for the maintainer as well as the user. Although if this is in a separate file, then you could have the "main" class documentation isolated from this.

I still don't get your idea. If _Decoder_K is included in the documentation (by removing the leading underline), then the methods (substitution, exponents, ...) of the class will be exposed. Those methods happened to be there just for implementation efficiency, and never intended to be used by a user. I think implementation details should not be exposed in the documentation.

Right they are the key workhorses. Maybe a good idea. Though I don't have much experiences with Cython files, I will try.

To Cythonize it, you just need to move your code into a .pyx. To make it a bit better, you add cdef before each class and create .pxd file (the .pxd is like a .h header file in C) with those class declarations. There should be a number of easily accessible examples for this; e.g., groups/group.pxd.

Done. I see 10% speed increase in doctesting by cythonization. Thanks.