Some cyclically permuted HMFs Hecke eigenvalues need to be recomputed

[jvoight]

This bug was found by @AurelPage and @JohnCremona. JV

[JohnCremona]

This occurs for 3.3.49.1 and several quartic fields, based on evidence I have gathered so far while computing elliptic curves attached to dimension 1 Hilbert newforms (or trying to). When the quartic fields are done I will post a list here of those where the data is suspicious.

[jvoight]

Hi @JohnCremona ,

I'm almost done (fingers crossed) recomputing the HMF data for the affected forms.

I found in total 620 (Galois conjugacy classes of) forms, occurring over the fields '3.3.49.1', '3.3.81.1', '3.3.169.1', '3.3.361.1', '3.3.961.1', '3.3.1369.1', '3.3.1849.1', '5.5.14641.1, that are Galois, odd degree, and with forms of squarefull level: these are the ones where in running the indefinite method and recombining with the definite method, some issue might have arisen. I think I isolated what the problem was--an annoying problem with code using Automorphisms() and rewriting primes--but I am not 100% sure about this (it's been too long, and I don't exactly remember what scripts I ran as these last minute patches).

Unfortunately, this does not give any insight into your missing elliptic curves, so at some point (for LMFDB v1.1?), we should dig into this more seriously.

For v2.0, I'd like to do HMF with higher (nonparallel) weight, and character, modelling after a stable version of the classical modular forms--this will come after some new algorithms. This promises some other mathematical rewards, and I have some other optimizations in mind. For that reason, I don't want to recompute the entire existing HMF dataset as it stands, or try to extend it using the existing implementation. (As far as I checked, I did not see any other problems, but of course I can't be sure. First, I do believe we will have more detection/certification of forms when we have the L-functions precomputed. Second, I'm also working with Sam and Ben to play with q-expansions of Hilbert modular forms, in which case this would give another way to sanity check the forms.)

Anyway, rerunning makes files like the one attached, similar to old data files. For safety, I'm computing all conjugate levels too, to match. data_3_00169_recomputed_squarefull.txt

Once everything is done, I will ensure that the eigenvalues are permutations of the old eigenvalues, so the alphabet labels remain the same. (For my own sanity, I will also probably replace matching lines in my text files and make a v1.2 of them.)

So my question to you: after that, may I use my old upload script to replace them in the mongodb? I think that should cause minimal damage--and I'll be careful as I go--but I wanted to make sure I'm not overwriting something that you've done.

[jvoight]

One other "heads up" for @davidfarmer about serving HMF L-functions from the database in v2.0: for HMFs I think I'd like to do something like compute all HMFs whose L-function has conductor bounded by some number, and then for each to compute enough Hecke eigenvalues to ensure that the zeros are correct to a fixed number of digits and the Z-plot is good in some range. For this, I'd want to work closely with someone who knows the details of the precision required, etc., to make a good plan. Is this you or Andy Booker or someone else? Did you all do something like this already for the classical modular forms? It would be nice if one was modeled after the other.

Anyway, a comment that can be ignored for now, but I wanted to get it down before I forget. :)

[jvoight]

My intention was to match up one-to-one all "bad" forms in the database (all 620, including conjugates) with the recomputed forms. So I don't think there will be any consistencies this way; but just to be safe, I'll look carefully over the upload script. JV

[JohnCremona]

I will also look at my new import script since I just rebased it on current master. We should certainly try this on the data for just one field to see how it goes. My import code also includes export code, the idea being that we should be able to create "corrected" import data files from what is in the database, since there are a few ways in which these have changed since the original imports. These changes (that I can think of) are (1) adding A-L eigenvalues; (2) adding conjugate levels; (3) corrected the letter codes for when there are >26 newforms at a given level (JV's original data files went from y,z to aa instead of ba; in the database these have all been corrected). The new import script was written before that change.

[JohnCremona]

JV: you might like to look at my branch called hmf-import. There's(1) a new version of import_hmf_data called import_hmf_data_new (which was originally just a copy so your old functions are in there too); (2) the file hilbert_field has been added to, so that you can create one from one of your data files as well as from the database, with a utility to check that a field (re)created from the file agrees in all respects with one already in the database. This class has been used already by Aurel and me but is not currently used in the production code; (3) a file webHMF which creates a new class for holding a HMF: it can be created from the database via a label, or from a data string in an import file, and can compare itself (after creation from such a string) with a database entry. Not yet implemented there is the function to store the HMF in the database. The idea is that the import script will use the two classes (one for a field and one for each HMF), so that almost all of the code now in the actual import* will be redundant. And then, the actual production code for fetching a HMF from the database an displaying it should also use the new class. Where I stopped working on this (summer 2015) was when I realised that your original datafiles did not contain the AL eigenvalues.

[jvoight]

Thanks John--this code was really useful. (I regret making certain decisions about various aspects of the HMF computation, but I supposed hindsight is 20/20.)

I had to modify your import script to my needs: Archive.zip (with a sample new data file)

I ran it on all of my data files under the test feature, and everything lined up (after a bit of tweaking for Galois). In several places where I could test for an elliptic curve, it is now found. But have a look and let me know if anything looks awry, but to make the deadline tomorrow (Sunday) afternoon EDT I'm going to run this to fix up the files.

[JohnCremona]

Wow, I did not know you wanted to fix these before the release. I'll take a look today (despite your most un-git-like zip file). When you are done (with each field) I can try to fill in gaps by running my elliptic curve finding script.

[JohnCremona]

I can add the new import file to the lmfdb code right away as it affects nothing else. The test data file is for the field of discriminant 81, right? I had no problems with that field, I think (I checked my logs). What am I looking for in that file? If you run the new import script on that file would you expect the data for that field to change at all? Sorry if I am being dumb.

On 1 May 2016 at 01:14, jvoight notifications@github.com wrote:

Thanks John--this code was really useful. (I regret making certain decisions about various aspects of the HMF computation, but I supposed hindsight is 20/20.)

I had to modify your import script to my needs: Archive.zip https://github.com/LMFDB/lmfdb/files/243886/Archive.zip (with a sample new data file)

I ran it on all of my data files under the test feature, and everything lined up (after a bit of tweaking for Galois). In several places where I could test for an elliptic curve, it is now found. But have a look and let me know if anything looks awry, but to make the deadline tomorrow afternoon EDT I'm going to run this to fix up the files.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/LMFDB/lmfdb/issues/444#issuecomment-216004262

[jvoight]

Yes, un-git-like, should I have included the datafile somehow in the repository (or to make another one)? Seemed weird to me.

Anyway, for the field I sent, most are duplicates, but in level norm 576, it finds several forms with permuted eigenvalues (3.3.81.1-576.1-...)

You're not being dumb, I'm sorry that I didn't explain well what I'm doing.

I think we're all set to go... WITH the exception of one super annoying huge space (3.3.49.1-1849.6-...), that is still computing and probably will require the whole week. If it remains wrong for the release, so be it. But I won't forget about it, and will fix it as soon as the computation finishes.

[JohnCremona]

OK, go ahead. I'll be offline for an hour or two now anyway.

No, the test data should not have been in the code repository, and sending that directly was fine! I have not yet put the new import script in either but there is no hurry for that.

John

On 1 May 2016 at 17:43, jvoight notifications@github.com wrote:

Yes, un-git-like, should I have included the datafile somehow in the repository (or to make another one)? Seemed weird to me.

Anyway, for the field I sent, most are duplicates, but in level norm 576, it finds several forms with permuted eigenvalues (3.3.81.1-576.1-...)

You're not being dumb, I'm sorry that I didn't explain well what I'm doing.

I think we're all set to go... WITH the exception of one super annoying huge space (3.3.49.1-1849.6-...), that is still computing and probably will require the whole week. If it remains wrong for the release, so be it. But I won't forget about it, and will fix it as soon as the computation finishes.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/LMFDB/lmfdb/issues/444#issuecomment-216054515

[jvoight]

Thanks John! I added the import script #1120. The mongodb has been updated, so this could be closed except for one final space (which I will update as soon as it finishes computing).

[JohnCremona]

@jv can you send me a list of updated fields? I retested 3.3.81.1 and found one where newform and curve with same label don't match up. On 1 May 2016 18:07, "jvoight" notifications@github.com wrote:

Thanks John! I added the import script #1120 https://github.com/LMFDB/lmfdb/pull/1120. The mongodb has been updated, so this could be closed except for one final space (which I will update as soon as it finishes computing).

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/LMFDB/lmfdb/issues/444#issuecomment-216055805

[jvoight]

OK, it turns out that the last form that I was waiting for (level norm 1849 over 3.3.49.1)--which took a stupidly long time to compute--was actually correct in the database already! So that means this issue is closed.

[jvoight]

Oops! Spoke too soon (used wrong file). This has been updated now (and I double-checked it).

@AndrewVSutherland : This amounted to a change of 12 entries in hmf.forms. It is not critical for them to be copied over to the cloud (it does not trigger a bug), so if there will be a routine copying of the entire database in the near future, it could be ported then. If we're still doing things one collection at a time for a while, then we should copy it over. (In principle, I could also just run the same script on the cloud, but I think that sets a bad precedent.)

[AndrewVSutherland]

@jvoight @edgarcosta We currently have no plan to do routine copying of the Warwick mongo db to the cloud (we will be rebuilding both mongo dbs, but that won't transfer data from one to the other), nor should we -- at any given point in time there is a reasonably chance that some collection is in the process of being updated by a script someone has running.

I will copy the hmfs.forms collection over (I have a script that will copy a specified list of collections in a given database from warwick to the cloud (actually from the replica m0 to ms)).

No one should ever update individual records on the cloud mongo db.

[edgarcosta]

@AndrewVSutherland @jvoight I am in the process of pushing the data of the development db to the lmfdb.org right now (as I had to take care of some technical stuff, swithing the storage engine, and this is the least human effort way to do those changes)

@AndrewVSutherland Don't do those changes yet, I will let you know once I replace the disk

[AndrewVSutherland]

@edgarcosta I will hold off. But are you sure we really want to move the whole development db to lmfdb.org? This is going to mean cleaning up a bunch of collections again, and make mongo db allocate several 100 GB more space than it needs. And are you sure that you have a quiescent snapshot of the development db that is in a consistent state? I know of at least 4 people that have been updating collections in various dbs over the past several days.

[edgarcosta]

@AndrewVSutherland I'm confused. Let's switch to email, as this has nothing to do with this issue.

[edgarcosta]

@jvoight Drew will take care of pushing hmfs.forms to lmfdb.org

Everyone else can ignore all my nonsense, as I had misinterpreted a conversation with Drew.

[jvoight]

OK, assuming Drew will push hmf.forms, I'm closing this issue for psychological reasons--it has required a lot of energy to fix!

[AndrewVSutherland]

@jvoight The collection has been pushed -- can you verify that all is well?

[jvoight]

Confirmed, these forms have been fixed.