Potential divide and conquer strategy?

[km-git-acc]

@rudolph-git-acc and @mariotrevi

Few days back, I had noticed in the fourth thread at Prof. Tao's blog you were focused on the Ceff term, and improving Ht precision at smaller values of x, while I have been working more on the medium values of x.

This could be a good divide and conquer strategy in light of the fifth thread. Do let me know if there are any thoughts on this, and if we can help each other more.

[rudolph-git-acc]

@km-git-acc

Happy to follow your suggestion and divide up the work. If there are any specific requirements you would like me to work on, just let me know. I do have the Pari/gp code working for A_eff, B-eff and E1,2,3 as well as their bounds. Today I have attempted to find a faster formulae to produce the exact Ht, say for x<10^3, however have not succeeded yet.

P.S. Did you see my comment from last Saturday on your E1/E2 bound calculation? It is just a small thing, but probably good to get the Wiki-displays for Sn+ and Sn- fixed.

Regards,

Rudolph

[km-git-acc]

@rudolph-git-acc I had seen your comment and your strategy of calculating N from x and then calculating the bound seems correct. Didn't completely follow you on the sn+ and sn- part. Is there a typo there?

EDIT: I replied on the latter part in the other thread. Although it would be good to ask Prof. Tao about this.

[rudolph-git-acc]

@km-git-acc

It indeed just seems a typo and as you say, we do need the extra divide by 2 in both sn+ and sn-. Will ask him directly on the blog.

Just saw that you published the bounds for the normalised derivative of A_eff + B_eff and I do fully match your numbers in Pari/gp code.

I also produced the code for the normalised derivatives of A_eff + B_eff and I guess you have coded this as well. So, would like to check some numbers that I got (way below the bound).

t=y=0.4

x= 10^2 0.19243854524130085276 x= 10^3 0.29955042110470048266 x= 10^4 0.028704733253812567905 x= 10^5 0.22145785748176529207 x= 10^6 0.48873623232165474190

Could you confirm?

In the mean time I have started to produce the exact values for H_t, in steps of 0.1 for x <= 300 at a precision of 40 digits. Have already produced up to the first 200 and expect the last csv-file to complete tonight. Maybe these could also be posted in the research folder.

[km-git-acc]

@rudolph-git-acc

I have been trying to produce the normalised derivatives for A_eff + B_eff, but going a bit slow. Since this is supposed to be exact, and we cannot use inequalities, I wanted to be a bit careful. For example, in the A_eff section, differentiation gives (−log(n)ddx(sA)+ddxlogλ), which turns into (log(n)ddx(sA)+ddxlogλ) due to the inequality. Just checking if there are any such other nuances.

[rudolph-git-acc]

@km-git-acc

Yep. I spotted that one as well. I believe he assumed that -log(n)d/dxSa +d/dxlog(lambda) = I/2log(n) +d/dx log(lambda), so that the entire LHS can be replaced by the O<=(...) inequality.

[km-git-acc]

@rudolph-git-acc Also, I noticed while calculating the bounds that =(−i/4)log(s/2πn)−(i/4)log[(1−s)/2πn] in the next step becomes (1/4)log(|s||1−s|/4πn^2], whereas I think it should be (1/4)log(|s||1−s|/4π^2n^2]. Please check on your side as well.

[rudolph-git-acc]

@km-git-acc

Good catch. This should change to π^2 in both the bound as well as in the derivative, right?

[rudolph-git-acc]

@km-git-acc

After the correction of the π^2 in both the bound and the derivative itself, I now get:

Derivative: x= 10^1 0 x= 10^2 0.19257287640109451000 x= 10^3 0.29955745401936522916 x= 10^4 0.028704710631748847242 x= 10^5 0.22145782200606050720 x= 10^6 0.48873623602067662246

Bound: N= 10^1 5.2342874398389429723 N= 10^2 6.1615758838756614645 N= 10^3 2.5190679826910524273 N= 10^4 0.73829280418351811203 N= 10^5 0.28681345856569107338 N= 10^6 0.13681914060945870938

And a few variables for x = 1000, y=t=0.4 in the derivative calculation:

Sa=500.15696362404976313 Sb=500.15763352746266180 ddxsa=0.50000001570603390126 ddxsb=0.50012532371741988033 ABsum1=0.29971404441875070455 Absum2=0.00086679552518089579374 lambda=0.41669127076771460641

[km-git-acc]

@rudolph-git-acc

For the bound, our values match.

I am getting different values for the exact derivative at x.

10^2, 0.5534567762349945175223569395 10^3, 1.0368922891066215403945020717 10^4, 0.654362756931538124392228692915 10^5, 0.862586325880511659690526817054 10^6, 1.41149560304156500014976609879

I have noticed one more nuance. In the triangle inequality, it did not matter whether s is chosen as 1+y-ix or 1-y+ix to derive ddx log_lambda. But in the exact derivative, if we use the former, then log lambda is actually -log lambda. and then the expression in the numerator would be −logn * ddx(sA) - ddxlogλ

EDIT: Have added the function ddx_abbeff_x for the exact derivative calculation in another branch https://github.com/km-git-acc/dbn_upper_bound/blob/Derivatives-for-eff-model/dbn_upper_bound/python/research/ddx_abbeff_bounds.py

May not be the final version, so havent merged it into the main branch yet. Once our values match, then that can be done.

[km-git-acc]

@rudolph-git-acc Also, as Prof. Tao commented just a while back, for the bound formula, there is an additional factor of 4 giving 16pi^2n^2, if we are dealing in x and y, and correcting that leads to further improvement. For the exact derivative, I was computing using s, so no change there.

[rudolph-git-acc]

@rudolph-git-acc

Yes, saw it. Not only impacts the bound, it Is also induced a change in the derivative calculation itself (same factor).

Currently running through the code in detail to spot the difference. Could you maybe post a few variables for say x=1000? Let's just use the corrected version (i.e. with 16 Pi^2).

[rudolph-git-acc]

@km-git-acc

Please ignore my comment on the 16*Pi^2. That just sits in the O<-() part. Symbols dancing in front of my eyes now :-)

[rudolph-git-acc]

@km-git-acc

Our data is very close now. Here is one difference I found:

• (I/4.0)log(sb/(2pi)) - (I/4.0)log((1-sb)/(2pi))

should be:

• (I/4.0)log(sb/(2pin)) - (I/4.0)log((1-sb)/(2pin)) (the n is missing)

Will keep digging further.

EDIT: ignore my comment. This is not an error in your code, I now understand how the n's got removed.

[rudolph-git-acc]

@km-git-acc

The differences are small, but I would really like to get it fully matched. These are my variables for x=1000. Could we compare, so we can zoom in?

alfap1 = 0.0010000302193660523824 ddxsa=0.50000066999756590754 ddxsb=0.50000074999804750530 ddxloglambda=1.0892570119404423856 (n=9) sumB=0.29963939014413037597 sumA=2.3543846706393712008 lambda=0.41669127076771460641 overall=1.0571656625195835368

Thanks.

[rudolph-git-acc]

@km-git-acc

Never mind, I have got an exact match :-) Although, I see again at 16 digits, the numbers start to diverge, so maybe there still is somewhere a non-mpmath variable in your code?

I had made two errors in the ddxloglambda. Mistook an "i" for a "1" and had a 1-s where it should have been an s.

Just saw prof Tao's comment on the blog that we should use the +-sign. However, the definition of s=(1+y-ix)/2 puzzled me as well, since usually we have s=(1-y+ix)/2. I'll now try to check how well it works against the Newton quotient.

[rudolph-git-acc]

@km-git-acc

Oops. I just saw how the I/2*log(n)-term actually removes the n's from (I/4.0)log(sb/(2pi n)) - (I/4.0)log((1-sb)/(2pi n)). Your code is correct after all (as expected) :-)

Per prof. Tao's suggestion, I calculated the Newton quotient with an h=0.00001 and:

for bn(−logn ddx(sA) – ddx(logλ)), we do get a very good fit

10^2 0.553456776234994527959785732693 (Pari/gp) 10^2 0.5534571139180834501 (Newton) 10^3 1.03689228910662160759493507488 10^3 1.0368883876331376612 10^4 0.654362756931538164406520908411 10^4 0.6543622036312176871 10^5 0.862586325880511759178842303919 10^5 0.8625847672806026730 10^6 1.41149560304156519002771266906 10^6 1.4114981150525967460

However using bn(−logn ddx(sA) + ddx(logλ)) yields a quite poor result: 0.169823651285985695837908766585 (Pari/gp) 0.5534571139180834501 (Newton) 0.711601905916954640803819249907 1.0368883876331376612 0.816016755000099945512485204904 0.6543622036312176871 1.25381761152826826418963000298 0.8625847672806026730 1.08876853050467835423059836262 1.4114981150525967460

So, it looks like your comment is probably correct.

[km-git-acc]

@rudolph-git-acc

Yeah, I think it turns out to be the same thing in practice. One has to be just careful with the signs. This is what I get on comparing the derivatives with the newton quotient (same spacing) x, ddx (current implementation), ddx_opp, newton quotient 1000.0, 1.0368922891066215403945020717, 0.711601905916954603964319255468, 1.03688838763101295033311342269 10000.0, 0.654362756931538124392228692915, 0.816016755000099877255766020804, 0.654362203662324180242447005411 100000.0, 0.862586325880511659690526817054, 1.25381761152826813205003362336, 0.862584766907228639547962907347

I think it will be too much effort trying to separate mpmath and python. As a general rule, what we can do is, 1) if our first many significant digits match, then our results are aligned (unless there is a strong reason to doubt otherwise), 2) whenever something requires a lot of precision for publishing, we should use the numbers obtained by Pari/GP.

These are the detailed numbers I got for x=1000.0

ddxsa 0.500000669997565907535416847657 ddxsb 0.500000749998047505304378685444 ddxloglambda 2.18852425891921819981378003767 sumB 0.2996393901441303653188408443 sumA 2.17802961261388198601688293173 lambda 0.416691270767714586163140957936 overall 1.0368922891066215403945020717

Also, once our numbers match (which they seem to now), please go ahead and publish your results on the blog.

[rudolph-git-acc]

@km-git-acc

Great, I do get a full match on the variables :-)

Have attempted to post the results on the blog, but it currently seems blocked for some reason. Have tried a couple of times and each time I press 'post comment', it just vanishes and doesn't appear on the blog. Will try again later.

[rudolph-git-acc]

@km-git-acc

I have now finished computing Ht for t=y=0.4, with an accuracy of 40 digits, for x from 0-300 in steps of 0.1. There are three files of 1000 entries in a txt-format (two fields comma separated) attached to this comment. Could you maybe put them in the relevant library?

Ht values y=t=0.4 for x= 0 to 100.txt Ht values y=t=0.4 for x= 100.1 to 200.txt Ht values y=t=0.4 for x= 200.1 to 300.txt

I aim to write a short program tomorrow to compare |(Ht - A_eff - B_eff)| with (|E1|+|E2|+|E3|) in steps of 0.1 for this low range of x and see how it looks.

[km-git-acc]

@rudolph-git-acc I have put them in the research folder as part of a sub folder. Here's the link https://github.com/km-git-acc/dbn_upper_bound/tree/master/dbn_upper_bound/python/research/H_t%20at%20small%20x

Is this through the integral calculation and hence exact? I think you should mention about the progress on this on the blog as well, as this is part of Goal 3.

[km-git-acc]

@rudolph-git-acc As per my latest comment on the blog, I am planning to start large scale computations. Do you want to divide the x ranges among ourselves? N=300 is approx 1.1 million, and the grid size is expected to be on average 0.4/6 ~ 0.06 at the start, but increasing afterwards. In terms of datapoints for each x, what should we aim for? I am thinking, (x, |f(x)|, exact |f'(x)|). Will retaining the complex valued f(x) and f'(x) be useful (since they take up a lot of space)?

[rudolph-git-acc]

@km-git-acc

Finally managed to post our results on the blog. For some reason I couldn't get things posted today and after I tried from my iPad, the post ended up in a "your comment requires moderation" state...

Anyway, got through now and prof. Tao has responded to the error on wiki (it was indeed the different definition of s).

I have indeed computed the 3000 values through the integral and used a sum of 1 to 9 for the super exponential series and for the integral a limit of 9. I checked for each run of 1000 steps whether the highest value of the batch i.e.: 100, 200 and 300 would remain stable at 40 digits when I further increased the limits and the accuracy (the latter seems to be the most important to get right).

On the large scale computations, I would be very happy to contribute and divide up the work. I have however no experience yet with these type of calculations and would like to follow your lead on this. I guess we would both need to use the same software for the computations? My gut feel is that we should store as much relevant info as possible, since we would most certainly regret it when we miss some data and would have to do the runs all over again :-)

[km-git-acc]

@rudolph-git-acc

I will share some code on how I run the loops and save the results. I think the only main addition is the ability to save results to a text file after every n (n could be 1000, 5000, or some other number) evaluations.

Yeah, we should probably save as much as possible. So x, N, c, fx, |fx|, f'x, |f'x| Where c can either be a fixed value - largest normalized error for N>=20, or we could vary c by N, which would make things faster, since c would decrease dramatically as N increases. Maybe we could just choose c to be 0.128, which is the bound obtained for large x (it would possibly increase the amount of calculations overall by 20-30%).

In terms of x ranges, I can cover x for N 151 to 300, and you would have N=20 to 150.

[rudolph-git-acc]

@km-git-acc

Sounds good! Just saw a comment from prof. Tao that he has an idea for a bounded function for the derivative of Ht that is valid at lower values of x. Is it wise to wait for that or would such a function only be valid in the domain for N < 20 anyway?

[km-git-acc]

@rudolph-git-acc

I think any calculation using a mesh will require computing both H_t and its derivative. So the main challenge with the direct approach would be not the bounded derivative of H_t, but the accurate integral calculation of H_t at larger x, which has always been a bottleneck. So the direct approach may only be feasible for N<20.

By the way, I have tried out parigp. Please check this file https://github.com/km-git-acc/dbn_upper_bound/blob/parigp/dbn_upper_bound/parigp/abbeff_eval.txt

I have verified the numbers, but please verify again before using. We can both run the same code but in different N ranges (to be changed in the last section). I am starting the run between N=151 to 300. Incidentally, even in the low range of N=20 to 28, with some random evaluations, I have never seen |(A+B)/B0| go below 0.07. The maximum normalized error for N>=20 is ~ 0.0647, so I have set c to 0.065.

Parigp does have good advantages - more concise expressions, and one doesn't have to worry about precision. But one challenge I experienced is that it's not flexible in how to save the output. So right now I have set saving of some text blob after every 5000 evaluations (basically a list of 5000 lists), which will later have to be cleaned up and separated into multiple lines using some regex. If let's say the list of lists during can be directly saved as 5000 different rows, that would be awesome. Also nested loops seems to lead to fickle behavior, so the inner loop is right now just one long line.

EDIT: changed the x-xNp1 condition in the while part to if(x>xNp1,0,1), since otherwise x-xNp1 would never exactly match and N wouldn't change.

[km-git-acc]

@rudolph-git-acc Also since I am operating in the higher ranges, I see very little chance of the estimate going below c at any point. But when you look at the code, can you add a way to break the inner loop if abs(est)<c. I tried something like if(abs(est)<c,break); which didn't work as expected.

[rudolph-git-acc]

@km-git-acc

The accurate integral calculation of H_t at larger x is definitely a problem. I already started to struggle for x > 300 (at 40 digits) and had to set pari's dps at 1000 digits. The integral calculation is very sensitive to the dps setting and requires a steep increase for each x + 100. Also the required computing time increases accordingly for higher x. An N of say 20 would imply an x of say 6000 and I saw that user Anonymous has posted an exact result for x=3000 and x=10000. Really curious how he or she has calculated the latter!

What I read from the code you shared, is that for each interval N and N+1, you establish the associated xN and xN+1. Then you run through this interval in steps of |(A_eff(x)+B_eff(x))/B0(x)| - 0.065) / (|d/dx (A_eff(x)+B_eff(x))| bound), until you reach (or exceed) xNp1. Steps are counted by 'i' and for each step in the interval you want to store the various fields in a file. Did I understand that correctly?

Two remarks:

• Just to be sure: is the y=0.4 pre-set in the procedure abbeff_x(x) and ddx_abbeff_bound(N) procedures?
• Is the use of a list really required? Is it for speed reasons? In pari/gp you could also try to write the output directly into the csv file for each incremental i (I believe it is cached).

Will try to get the code integrated into my own abbeff_x(x) and ddx_abbeff_bound(N) procedures, so we could compare output. Will also check the halting requirement abs(est)<c!

[rudolph-git-acc]

@km-git-acc

Attached the output of my first test run for N=151 (no for loop activated). I put a break in the while loop with the condition if(abs_ddxest>1,break) and the program correctly halts.

Curious now whether we got the same data :)

test.txt

[km-git-acc]

@rudolph-git-acc

Yeah, you have described it correctly. I compared the output of the first 5 rows in the test file with my output and they are exactly the same, except for formatting differences

• My precision is set to 30 digits, so the output is longer
• I am also saving the complex valued (A+B)/B0 and its derivative, not only their absolute values.

Also, its great you can use break procedures. I assume the condition on ddxest was for illustration. Does it also work with the abs(est)<c condition? This is unlikely to get invoked in practice even for N near 20 with c=0.065, so a way to check it would be to temporarily set c to a artificially high value like 0.8 or even 1.2, and to then check whether the condition is invoked.

I have completed the run for N=151 to 200. Reaching 300 may take a day. I just realized that N=20 to 150 is a much smaller range in terms of x, so your run will get completed earlier.

Yeah, we can pass y values to most of the procedures in the file. But i am assuming a default value of 0.4 (overridden if y is passed) if y is not passed. Similarly for t, although I had to again define it explicitly at the bottom for the xN calculation.

I always use a list approach, since I believe too many writes (one per evaluation) causes slowdowns and also affects the physical memory device. But it's possible I am being paranoid about this. Also, if parigp is doing some buffering internally and minimizes the number of writes, then doing that again may not be required.

[km-git-acc]

@rudolph-git-acc

This is my output for the first five rows of N=151 (copied from excel so the precision shown will be unequal).

test_km.txt

[km-git-acc]

@rudolph-git-acc On the other hand, for the task at hand your format is better. I am planning to initially share the file only with N,x,|f(x)|,|f'(x)|, and share the larger file with complex values only when needed. All the meta information such as c and the ddxbounds depending on N can be described or linked to. Also in this smaller file, I have reformatted the columns (except N) to always show upto 16 digits (including leading 0s). Since |f(x)| and |f'(x)| both are of order of magnitude 1 in our x range, although |f'(x)| has gone as low as 0.11, it does not cause a meaningful loss of information, and almost halves the file size.

[rudolph-git-acc]

@km-git-acc

Have gotten an exact match with your data :)

The halting condition is also correctly invoked, however I noticed that just changing the c-parameter is tricky to test it, since it also impacts the adaptive mesh calculation itself and thereby induces a very different set of numbers. When I put e.g. if(absAB<0.58,break) in the code itself, it correctly halts.

Let's stick to the previously agreed set of fields (I will also include the complex valued (A+B)/B0 and its derivative) and set the precision to 30 digits. We can indeed always derive a simpler format from this rich dataset.

I am now trying to produce the file for N =1=20..150, however have hit a strange error each time the N jumps up in the for loop. Will hopefully sort that out soon.

[km-git-acc]

@rudolph-git-acc

Great.

Indeed I first created the file in the originally agreed format, and then derived a simpler one out of that. Since GitHub won't allow such large files, it will have to be shared through other means (I am planning to share a Google drive link).

Also, my N range is almost complete and I am double checking on some random x points. Will share the link soon.

[km-git-acc]

@rudolph-git-acc

I had also run another test on N=11 to 19, but didn't complete it, with c=0.165. It seems this range can be covered as well using this approach. You may want to include that in your results. Then the direct evaluation will be hopefully faster (but probably still long enough), since x only up to 1520 will have to be checked.

[km-git-acc]

@rudolph-git-acc

I have uploaded the smaller file, and it can be accessed at https://drive.google.com/open?id=1NvEv-1R4KTEchWMbJCpZA6Uf1xLkiYWM Have also added a meta description explaining the file structure and basic definitions.

Will now upload and share the larger file from which this was extracted.

EDIT: Also sharing the link (only in this thread for now) for the much larger file with complex valued outputs and adhering to the original format (although i have eliminated the first column i).

N, ddxbound, c, x, est, abs_est, ddxest, abs_ddxest '151', '5.28211193705137432221688287699', '0.0650000000000000000000000000000', '286525.502218738143541423080864', '0.690704265972946872486184610560 - 0.0244613886800442041072916358462I', '0.691137281999302602783018646870', '0.292342056956613758596079635809 + 0.124096047324850372536233397732I', '0.317590470932733600789130612186'

https://drive.google.com/open?id=1T0f_kJk9aGk4F3hi0ghxPLcoDp84kQ3f

[rudolph-git-acc]

@km-git-acc

Nice work!

I have just completed the run for N=11 to 19 with c=0.165 (I guess that wasn't a typo in your post above?). The output file is 8.7 Mb in size. Let me know how you would like to receive it.

For a cross check: the last 3 lines are:

27280, 19, 5.68727234154094722762098872328, 0.165000000000000000000000000000, 5026.86488371541939115087268912, 0.519256511453725317518906285778 + 0.110863669057948368194397292320I, 0.530959582081426486317983890890, 0.415974842908784746334522715776 - 0.399790240583880666214610586535I, 0.576946536863776773167349843627

27281, 19, 5.68727234154094722762098872328, 0.165000000000000000000000000000, 5026.92923083300129175207983166, 0.547121911170062588441154474483 + 0.0885261712372872111710696156903I, 0.554237556176334332776324020895, 0.446338160595609237969561515483 - 0.293846280569811651779797916192I, 0.534381315362526809726665344196

27282, 19, 5.68727234154094722762098872328, 0.165000000000000000000000000000, 5026.99767094502709843815319233, 0.578065306232720271652964520469 + 0.0723305969902294771609400540482I, 0.582572925504517124951148476073, 0.453620410806987494418720858270 - 0.179619817739413801467703569061I, 0.487888056858784287435439248979

Running the N=20 to 150 now :-)

[km-git-acc]

@rudolph-git-acc

A minor thing, but the floor and ceil for xN and x(N+1) in the last section are not necessary. They were put for some testing and I forgot removing them later. They cause a small tail on both sides of an N in the output, which I removed manually later (although they are very unlikely to change any stats derived from the output).

[rudolph-git-acc]

@km-git-acc

That's no problem. I'll remove the ceil and floor and do rerun just to be sure. The 11 to 19 completed in less than 5 mins. You are sure about the c=0.165 instead of 0.065 for this domain, right?

EDIT: 11 to 19 done in < 2 mins (without the ceil and floor)

[km-git-acc]

@rudolph-git-acc

The output for N 11 to 19 matches. C was set higher since |E/B0| at N=11 is around 0.16.

I think this file size could be attached in the comment. Or you can create a Gdrive link out of it since it has to be shared with everyone anyways.

[rudolph-git-acc]

@km-git-acc

The output without the ceil and floor has changed quite a bit now. All numbers are different and the file also got slightly shorter. The last three lines now read:

27213, 19, 5.68727234154094722762098872328, 0.165000000000000000000000000000, 5025.91094837541814264609542868, 0.739763902854528131086304900537 + 0.542423547022115893115123844535I, 0.917318884756341046162392747382, -0.707376053883536880985631991876 - 0.0206506653013383807227068716441I, 0.707677420570440677302539937393

27214, 19, 5.68727234154094722762098872328, 0.165000000000000000000000000000, 5026.04322951821857741231545582, 0.646776650165231064099544222621 + 0.521952559790909392989056825170I, 0.831116423776621402109945351003, -0.680172286190245975983600614912 - 0.286401194530169661223099668642I, 0.738010828599129676134096626497

27215, 19, 5.68727234154094722762098872328, 0.165000000000000000000000000000, 5026.16035357765761236583232020, 0.572861968962891718461315896663 + 0.475905942473278959749340180786I, 0.744753181641690616487121084233, -0.569610504776649759050393338242 - 0.492760173055250172802717672264I, 0.753172433976011788285325327789

The first three lines now are: 1, 11, 4.79583167739456145443788613516, 0.165000000000000000000000000000, 1520.21668507210094809207313317, 1.07183965568936106411242333080 - 0.275265375505582571936074143252I, 1.10662155882691778213999001711, -1.07518747289589617442096961703 - 0.318886488603921016190044056654I, 1.12147968973423772781899256511

2, 11, 4.79583167739456145443788613516, 0.165000000000000000000000000000, 1520.41302673339513135969264146, 0.877560520986509381217568365779 - 0.400974050658633616629518810757I, 0.964827786341016533428906034186, -0.853179902800015790704653521358 - 0.951664920913066747577392603803I, 1.27811660979674228622687999865

3, 11, 4.79583167739456145443788613516, 0.165000000000000000000000000000, 1520.57980235025238106075985720, 0.767260258145151296935915018461 - 0.598461052794949443475694798272I, 0.973059060613179569905937444128, -0.437490367676640095362175788403 - 1.39465896961282827253686911614I, 1.46166735727776228929997127250

Will triple check whether I did something wrong!

[rudolph-git-acc]

@km-git-acc

Have rechecked everything and the numbers with and without the ceil/floor remain very different. I now completed the N=11 to 19 file (without the ceil and floor) and attached the output. The N=20 to 150 will be ready tomorrow morning EU-time (now at 60 within 25 minutes).

I did some further comparisons between the output with and without the ceil/floor functions and noted that when these functions are included, the x-values at the jumps of N are not smooth (i.e. each new N starts with an x that has no decimal digits), where the version without the ceil/floor does have smooth x-values at these jump points.

output N 11 to 19.txt

[km-git-acc]

@rudolph-git-acc

I have run multiple checks on the attached dataset and it is correct. (The earlier dataset using the floor and ceil was also correct with slight modifications, but its better to use the new one directly).

Essentially, the exact mesh points at which we generate the estimate is not important, as long as the mesh is fine enough. The way I understand it, for a non mesh point, we are using Taylor's expansion with the estimate at a mesh point and Lagrange remainder bound for the mesh interval to lower bound the estimate for the mesh interval.

So for any h<mesh size at an x, |f(x+h)| > |f(x)| - Mh, where f(x) = ((A+B)/B0)(x,y=0.4,t=0.4) in the test problem case where M is upper bound on the first derivative of f(x). So if |f(x)| - Mh > c (our desired lower bound for f(x+h)), then we have established |f(x+h)| > c for the entire interval (x-h,x+h), although in our procedure we only care about the interval (x,x+h), since the left subinterval is covered by the previous mesh point. And we keep c higher than |E/B0| so that |(H-A+B)/B0| < |E/B0| < c < |f(x+h)|, thus meeting our goal.

The file without using floor and ceil will have different mesh points than the file using them, because the mesh points the way they are calculated will always depend on the starting x for an N. But they both suffice for our purposes (the first one directly, and the second one if we finally remove those few points where for an N, x less than xN or x greater than x_(N+1). The second one after the removal of those few nuisance points will also have smooth x values.

For a sanity check, I checked whether the x values generated ever decreased compared to the previous x value (they did not, indicating |f(x|>c for all evaluations), and what was the minimum |f(x)| (around 0.31, much higher than c=0.165)

[rudolph-git-acc]

@km-git-acc

Many thanks for your clear explanation of the mesh-mechanism. Very helpful!

The file for N=20 to 150 is now complete and 595 Mb in size. Will try to make a Google Gdrive link (struggled a bit since the Gdrive service will be renamed per 3/12) and share it. Would you like me to rerun N=151 to 300 as well (without the ceil and floor)?

Give the strong results, I actually wonder whether there is room to run similar mesh-exercises for N < 11. Even maybe for the range x=100 to 1000, for which we now have actual Ht data in steps of 0.1, I produced some plots to check the smoothness of Ht and the wave-lengths of its oscillations are way larger than 0.1. It therefore seems that the mesh-approach would still work at very small N. Or do we get into trouble with the size of the E-terms/bounds in this domain?

[km-git-acc]

@rudolph-git-acc Well, I had removed the floor and ceil tails manually for N=151 to 300, so it's actually not necessary. No harm in running it if you want to, although it's a long computation and I feel our compute resources will be better used right now to tackle the remaining part directly with very small x, and soon achieve the magic number of 0.48!

I think there may be potential to run the current mesh exercise for N=10, but the error |E/B0| increases dramatically in this region, so I think it's better to be cautious here. For eg. see the last column of this table https://github.com/km-git-acc/dbn_upper_bound/blob/master/dbn_upper_bound/python/research/bounded_normalized_E1_and_E2_and_E3_and_overall_error.csv

[km-git-acc]

@rudolph-git-acc I think even for N<11 or 10, the concept of the mesh will remain, but the AB estimates won't be used. Instead there will be a direct attack on H_t, with (assuming z=x+0.4i) the aim of proving

|H_t(z+h)| > |H_t(z)| - Mh > some comfortably non zero c value. where z will be a mesh point, and M the upper bound of the H_t derivative that Prof. Tao is planning to derive. If M turns out to be low enough the integral based data you had generated earlier can potentially be reused.

[rudolph-git-acc]

@km-git-acc

In the context of meeting the magic number of 0.48, I wondered about the following: we probably will be able to show that Ht doesn't vanish for y=t=0.4, however does this automatically imply that it also doesn't vanish for t=0.4, y > 0.4 ?

[km-git-acc]

@rudolph-git-acc

I guess that is some contour integration and other complex analysis magic that Prof. Tao has mentioned about, if not already done rigorously.

[rudolph-git-acc]

@km-git-acc

I think I have figured out the Google drive service now and here is the link to the output for N=20 to 150. Does it work?

Would you be so kind to condense this into a summary file (like the one you did earlier and check the min, max etc) and then publish the results on the polymath blog?

https://drive.google.com/open?id=1ZBX7jNGXhQZQ50t8UX4boTPW93dmRAun

[km-git-acc]

@rudolph-git-acc No worries. I will do that with both this one and the N=11 to 19 one.

[km-git-acc]

@rudolph-git-acc Have you checked this wiki page http://michaelnielsen.org/polymath1/index.php?title=Bounding_the_derivative_of_H_t Only the F' part is there yet, but it should give an initial idea of the derivative bounds involved. Will be trying it out.