WD-40 Issues thread was accidentally deleted by @keean

[shelby3]

I will continue the discussion here, until or if @keean can get Github to restore the thread. Luckily I still have a copy of the deleted #35 thread loaded, so I can copy and paste from it. If the original thread is restored, I will copy my posts from here back to that thread and delete this thread. I will give you an opportunity to copy your replies also before the deleting the thread. Perhaps @keean is the only person who can delete an issues thread.[or perhaps we’ll just link the old thread to this new one, because that thread was getting too long to load anyway]

@keean

I can find a few references to automatic memory management and GC vs ARC, but it seems not widespread. To avoid confusion we should probably use MS (mark sweep) and RC (reference counting) are forms of AMM (automatic memory management), and avoid use of GC altogether.

I find the term and acronym GC to be more recognizable than AMM which is why I continue to use it. And nearly no one knows the acronym MS. And you continue to use RC which widely known as radio-control and not ARC which is the correct term (employed by Rust docs for example) for automatic reference counting, because there is such as thing a manual reference counting.

IOW, memory has no more life if it can’t be referenced anymore and it continues to have a life for as long as any reference can access it. Whereas non-memory resources live on beyond (or need to die before) the life of the reference to the handle to the resource. The reference to the file handle is not isomorphic with the life of the resource referred to by the file handle.

You can always refactor so the life of the handle is the life of the resource.

And you completely failed to respond to my holistic reasons why doing so (for the multiple strong references case) would be problematic. And I grow extremely tired of this discussion with you because I have to repeat myself over and over ahead you repeat the same rebuttal which ignores my holistic points.

So am just going to let the argument with you stop now on this point. I do not concede that you’re correct. And I do not think you are correct. I already explained why and I will not reply again when you reply again totally ignoring and not addressing my holistic point. Total waste of my time to around and around in a circle with you making no progress on getting you to respond in substance.

If the resource needs to be shorter lived you can use a Weak reference. If the resource needs to be longer lived you can store it in a global container (global array, hash map, singleton object etc).

Which as we have already agreed is being explicit and which is what I wrote yesterday. But yet you still side-step my holistic points. Yes you can do the same things with ARC that we can do without ARC in the weak references case, but that does not address my point that in that case ARC is not better and is arguably worse because it conflates separate concerns.

ARC is for freeing access to a reference (thus it follows that the memory resource can be freed because no reference can access it anymore), not optimally semantic congruent with freeing (non-memory) resources that reference used to know about. They are not semantically equivalent.

There is no reason for them not to have the same scope. If I have access to the handle, I should be able to read from the file. If I don't need access any more destroy the handle. There is no need to have a handle to a closed file.

I guess you did not assimilate what I wrote about breaking encapsulation. But nevermind. I don’t wish to expend more verbiage trying to get you to respond to something you wish to ignore and skip over with addressing it.

Also I am starting to contemplate that the encapsulation problem is fundamental and we need to paradigm-shift this entire line of discussion to a new design for handling resources (but that will come in a future comment post).

Even GC + implicit destructors if we use static escape analysis with a single explicitly typed strong reference.

No, because a strong reference must never be invalid, so you cannot close a strong reference to a file.

You’ve got your model inverted from my model. I noted that in one my prior responses.

In my model the “strong finalizer reference” controls when the resource is closed. And the weak references are slaves. When you tried to fix my example, you did not, because you used a weak reference but I wanted the resource to actually be released before the asynchronous event. Thus I wanted to close the strong reference. The static linear type system can ensure no access to the strong reference after calling the finalizer. Thus it can also work with non-ARC. (However as I alluded to above, I do not want to recreate Rust and thus I am thinking about paradigm-shifting this entire line of discussion in a future post).

A strong reference to a file must be RAII.

Nope as explained above.

Basically with a strong reference you must always be able to call read on the handle without an error.

By definition, if you want to be able to explicitly close the handle, it is by definition a weak reference to a file.

By your definition, but I had a different model in mind.

So with GC the close is explicit, the weakness of the handle is implicit. You have no choice over this (unless you want unsoundness).

I explained it above. Open your mind.

Well we could even have a single explicit strong reference (thus any weak references are implicitly typed by default, although of course access to a weak reference always requires either explicit conditional test or an implicit runtime exception when use-after-destructed/use-after-finalized) with implicit (or explicit) destructors and make it work with either ARC or non-ARC, GC. Thus I see no advantage to conflating with ARC? And I do conceive disadvantages to conflating it with ARC, not only including that it won’t work with non-ARC, but also because conflating with ARC encourages the programmer to conflate reference access lifetimes with resource lifetimes which are not semantically isomorphic.

To repeat above you should not use a strong reference to the resource with GC, because that would rely on finalizers to release the handle, and that can lead to resource starvation. It's not safe.

To repeat you do not seem to understand what I explain.

Edit: Regarding C++, yes you are right you would swap a null there, but that's C++, which is not an ARC language. This would imply that "Weak" is defined:

type Weak<T> = T | null

And therefore Weak would be a billable reference to the strong file handle. However you would not be allowed to just write a null. Weak is an abstract datatype, so the value is private, and you would have to call weak.delete() which calls the destructor on the object contained, and then replaces with a null.

Okay but it doesn’t not rebut any of the context of my point which I will quote again as follows:

I was referring to the implicit release of the resource upon the destruction of the strong reference in the Map of your upthread example code, which is conflated with your code’s explicit removal from the Map (regardless of how you achieve it by assigning null or explicit move semantics which assigns a null and destructs on stack frame scope). You’re ostensibly not fully assimilating my conceptualization of the holistic issue, including how semantic conflation leads to obfuscation of intent and even encouraging the programmer to ignore any intent at all (leading to unreadable code and worse such as very difficult to refactor code with insidious resource starvation).

[shelby3]

@Ichoran

Um, your supposed code counterexample isn't a counterexample. In fact, the disproof of it being a counterexample was in my Rust example! Here it is again:

https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=2baade67d8ce2cc9df628de2b753f0e6

Here is the code, in case you want to see it here instead of run the runnable example there:

struct Str {
    string: String
}

impl Drop for Str {
    fn drop(&mut self) { println!("Dropping!"); }
}

fn main() {
    let mut s = String::new();
    s.push_str("salmon");
    let ss = Str{ string: s };
    let l = ss.string.len();
    std::mem::drop(ss);
    println!("I was size {}", l);
}

Rust drops at block closure, but it resolves lifetimes to last use when needed ("non-lexical lifetimes"), so it could do it automatically at the point where it says std::mem::drop.

So your first example would magically just work. If not, the explicit drop with move semantics (so you can't use it again) would non-magically work.

Seems you completely ignored what I wrote earlier in the thread in response to @sighoya:

If there is only one reference (or at least only one strong reference) to the resource handle then the compiler should be able to reason (by tracking the implicit linear type) when that reference dies either upon exit from its terminal stack scope or assignment of null, i.e. Rust does this but hopefully it can be accomplished without lifetime annotations? (If not, then @Ichoran may be correct to cite that as a utility advantage for Rust’s borrow checker) With multiple strong references to the same resource handle I presume the static analysis required by the compiler would require total, dependent type knowledge a la Coq and thus wouldn’t be viable for a general purpose PL. For multiple strong references I think only a conflation with ARC would be viable, but I have posited that comes with other meta-level baggage.

---

In contrast, RAII handles this flawlessly.

No it doesn’t. It obscures intent which can lead to inadvertent resource starvation. Yet I repeat myself again.

And Rust is not doing the ARC implementation of the form of RAII which can return the resource from the stack frame scope (i.e. not the “basic use case where the lifetime of an RAII object ends due to scope exit”), it is entirely a static analysis which is what I was discussing with @sighoya.

AFAIK Rust can’t handle the multiple strong references case, although I’m arguing against allowing (or at least discouraging it) it anyway. But surely C++ allows it via std::shared_ptr.

Having to remember to deal with resources is a pain. If the compiler can figure out that they need to be closed, they should just plain be closed.

If programming was only so simple and one-dimensional as you want to claim here on this issue.

If you need help understanding why your life is so easy now, then an IDE could help you out.

I actually find they often get in my way. It’s usually better to paradigm-shift than pile complexity on top of complexity.

[keean]

@shelby3 my use of Weak/Strong is in line with the common usage, for example see weak references in Java and JavaScript. Suggest we try and stick to the standard terminology.

The RAII case is less likely to lead to resource starvation because the programmer cannot forget to free the resource, and the resource is freed as soon as the program holds no references to the resource.

There is no semantic conflation. If you have the handle you can read from the file. If there is no handle you cannot read from the file. It's simple and clearly correct. The case where you can have access to a handle that is closed is far more confusing to me.

Don't forget that by wrapping that handle in a weak reference you can explicitly destroy it at any time. The purpose of a weak reference is to encapsulate something that has a sorter lifetime than the reference.

So from my point of view the explicit distinction between strong (the default) and weak references is the important thing. If we have this then we can have RAII file handles, and can use them in all the ways you want. I don't really think there is any conflation here, just a simplification the prevents a whole class of mistakes (unless you explicitly make the reference to the handle weak).

[shelby3]

@keean

my use of Weak/Strong is in line with the common usage, for example see weak references in Java and JavaScript. Suggest we try and stick to the standard terminology.

No I suggest you assimilate what I write and understand that I specifically wrote (much earlier in the discussion) to not conflate it with (the traditional meaning of) strong and weak references.

There is no semantic conflation. If you have the handle you can read from the file. If there is no handle you cannot read from the file. It's simple and clearly correct. The case where you can have access to a handle that is closed is far more confusing to me.

As I predicted more blah blah from you repeating your same myopic argument for the Nth time and totally failing to address my holistic point.

The last use of the reference to a data structure which somewhere in the object graph references a resource handle (and especially opaquely if you allow cyclic references) doesn’t correspond necessarily to end of the last access of the resource handle or the resource.

Don't forget that by wrapping that handle in a weak reference you can explicitly destroy it at any time.

Which in your model does not release the resource. Again that is tied into my holistic analysis, but it has become too complex (requires too much verbiage which I am unwilling to continue) to untangle all your vacuous non-rebuttals.

So from my point of view the explicit distinction between strong (the default) and weak references is the important thing. If we have this then we can have RAII file handles, and can use them in all the ways you want.

We can have RAII or even slightly better than RAII with a variant of my desire to ask the programmer to be explicit to make sure intent is not obfuscated or forgotten. And we do not even need ARC but that leads us to something like Rust, which I don’t like. And we already discussed that ARC is incompatible with non-ARC, GC, and I don’t like conflating resource lifetimes with reference lifetimes anyway (for the reasons I had already stated including obfuscating intent and explicit duration of lifetimes, e.g. where delete an item from a Map causing a resource lifetime to implicitly end and that is just the tip of iceberg of the implicit cascades that can occur causing insidious bewilderment as to why there is a resource starvation), especially not implicitly for the reasons I have stated which apparently you and @Ichoran have ostensibly not understood (or do not agree with and have not articulated to me why).

I don't really think there is any conflation here, just a simplification the prevents a whole class of mistakes (unless you explicitly make the reference to the handle weak).

Even with a single strong reference (and even if no weak references) and given implicit RAII (whether it be via ARC or linear types a la Rust) there is still a conflation of resource lifetime semantics with (reference or access to) resource handle lifetime semantics.

[keean]

@shelby3 I think you just don't understand some things that are common knowledge in programming and it's frustrating trying to explain them to you.

I can understand you have a preference for a particular style of semantics, say GC with explicit destructors, and that is fine. You should also understand that there are other people who prefer other idioms like RAII.

Your criticisms that there is something wrong with RAII, or use cases it cannot cope with are wrong. I think because you don't like it, you have not really studied it looked at how it solves the problems.

I have written programs using RAII, so I know how it helps avoid bugs. I have directly seen the benefits in real projects compared with explicit destructor approaches. So the situation from my point of view is that real world experience contradicts your speculation.

[shelby3]

@keean

I think you just don't understand some things that are common knowledge in programming and it's frustrating trying to explain them to you.

That is an extremely unfair allegation when in fact I wrote earlier in the discussion that I was making a distinction from the common usage of the terms. Do I need to quote it for you?

There is nothing wrong with the logic of my model. And you put your foot in your mouth (and I use that terminology because you were forcefully stating I was wrong) because you did not read carefully what I wrote. And so now you ’backsplain by attempting to claim that I don’t understand when in fact I wrote earlier on that I was using a different model.

You should also understand that there are other people who prefer other idioms like RAII.

What part of the following can’t you read?

We can have RAII or even slightly better than RAII with a variant of my desire to ask the programmer to be explicit to make sure intent is not obfuscated or forgotten. And we do not even need ARC but that leads us to something like Rust, which I don’t like. And we already discussed that ARC is incompatible with non-ARC, GC […]

What part of RAII on non-ARC requires (something like) Rust do you not understand? Even I wanted implicit I would have to use Rust if I don’t want ARC. Why do you conflate everything into a huge inkblot instead of understanding the orthogonal concerns I explain.

Did you entirely forget that I wrote that my new ALP idea does not use ARC?

I can understand you have a preference for a particular style of semantics, say GC with explicit destructors, and that is fine.

My preference for the programmer to express explicit intent was not about explicitly calling a finalizer in every case. You completed failed to follow the discussion from start to finish. I was advocating an explicit indication when something like RAII was going to be used in each instance where the reference is initially assigned.

Your criticisms that there is something wrong with RAII, or use cases it cannot cope with are wrong. I think because you don't like it, you have not really studied it looked at how it solves the problems.

I have written programs using RAII, so I know how it helps avoid bugs. I have directly seen the benefits in real projects compared with explicit destructor approaches. So the situation from my point of view is that real world experience contradicts your speculation.

I used ARC also for resource destruction in C++. But I also did not have a highly concurrent program. And I wasn’t doing anything significant that would have significantly stressed resource starvation if my usage was suboptimal.

But the points I made against implicit RAII were not that it can’t be made to work and were not a claim that it isn’t convenient and doesn’t prevent bugs compared to an unchecked manual cleanup (which I never advocated completely unchecked manual cleanup!), which just goes to show you have not even bothered to really understand what I wrote.

My points are about the obfuscation in code it creates and how that can lead to suboptimal outcomes and unreadable code in some cases. Surely you can fix bugs and mangle your code to make it all work, but you may make it even more unreadable. My point was an attempt think about how to make it even better and how to solve the problem of ARC being incompatible with non-ARC, GC (as we agreed merging the two would bifurcate into a What Color is Your Function) and without having to take on the complexity of Rust’s borrow checker. While hopefully also gaining more transparency in the code.

You completely mischaracterize what I wrote after deleting the entire thread where I wrote all of that.

[keean]

@shelby3 I have not said that your model is wrong, I am sure it's probably correct, just your use of non-standard terminology make it too much effort to decode.

I think your criticisms of RAII amount to a personal preference, and you don't seem to appreciate the real word benefits that I, and some others have been trying to explain to you, and are rooted in practical experience not speculation.

What part of RAII on non-ARC requires (something like) Rust do you not understand? Even I wanted implicit I would have to use Rust if I don’t want ARC.

Well you can do RAII in C++ so that's not quite right, but I get what you are trying to say, and I agree with you. I don't think you have understood what I am trying to say, because I have not disputed this.

[shelby3]

@keean

I think your criticisms of RAII amount to a personal preference, and you don't seem to appreciate the real word benefits that I, and some others have been trying to explain to you, and are rooted in practical experience not speculation.

Have I ever stated I don’t appreciate the benefits of RAII as compared to manual unchecked explicit release of resources? No! If you think I did, you simply failed to read carefully.

What part of RAII on non-ARC requires (something like) Rust do you not understand? Even I wanted implicit I would have to use Rust if I don’t want ARC.

Well you can do RAII in C++ so that's not quite right,

You did not even understand what I wrote! Hahaha. I was stating that I would have to punt to something like Rust only if I wanted to achieve something like RAII in a non-ARC scenario. To any extent that C++ is not using ARC, then it is employing move semantics like Rust.

but I get what you are trying to say, and I agree with you. I don't think you have understood what I am trying to say, because I have not disputed this.

I don’t think I have imputed that you disputed that claim ­— rather that you do not seem to incorporate in your understanding of what I am writing about, that claim is one of the factors in my analysis. There is a logical distinction. Discussion with me can include some mind bending orthogonal concerns, and it is difficult to keep up apparently. I think I know what Bill Gates felt like talking his interviewer as I cited before. Maybe the blame could be put on me for not having a fully threshed out replacement for Rust or ARC and having it all written up in a tidy blog. But I never thought that discussion had to be only for the stage where ideas were fully formed and hatched.

[Ichoran]

@shelby3 - Can you please just write a code example that actually truly genuinely shows your point that resource starvation is a likely outcome? Your existing example was disproved before you even posted it (whether or not you mentioned something to someone else later) because Rust lets you do what you said you couldn't do, and the resource is cleanable (if not clean because of Rust's drop-at-block-boundary semantics).

Like @keean I have spent a lot of time with RAII-based languages (C++ and Rust) and a lot of time with GC-based languages with using (mainly Scala).

When it comes to correctness in handling non-memory resources, empirically my observation is that it is far easier to do it in C++ and Rust. It's a constant problem with my Scala programs. It's basically never a problem in my Rust programs. (In C++ I had so many other problems that I can't recall how bad it was, just that various other things were worse.)

When it comes to explicit vs. implicit for things like this, I don't have a language that catches every case and insists that I be explicit about it, but I do have experience with parallel constructs (e.g. be-explicit-with-all-types, be-explicit-with-all-exceptions-thrown, be-explicit-with-all-returns) and in every case being explicit is a drawback. The reason is that attention is finite, and it wastes your attention on the obvious stuff that is working right every time (or which you can't even tell is working right because you're not a computer and can't trace every logical path in milliseconds like it can). Even saying raii val handle = new Handle is an extra hassle, an extra bit of type information that actually isn't useful in the vast majority of cases.

In addition to the extensive experience that this is the better way to do things (at least for someone with my cognitive capacities), there is also the critical invariant that @keean keeps mentioning over and over again, which you never have adequately and directly responded to, which is that RAII can prevent use-after-free (and use-before-initialization) errors.

So, write a code example that shows what you mean! So far you've failed every time. Everything is trivially fixable or isn't even the problem you claim it is. If it takes design to get it right, you additionally need to argue that this is harder than the alternative to get your proposal right. (So write two examples if you must in order to show the contrast.)

I am certain I am not understanding your objections without code; it seems as though @keean is also not.

Now, if your objection is just, "Rust does this all correctly, but I don't like Rust," then I understand that, but you keep saying things about risking resource starvation when my experience, and the logic of can-use-it-when-you-can-refer-to-it-can't-when-you-don't both argue that this is the way to avoid it.

[shelby3]

@Ichoran

Your existing example was disproved before you even posted it (whether or not you mentioned something to someone else later) because Rust lets you do what you said you couldn't do, and the resource is cleanable (if not clean because of Rust's drop-at-block-boundary semantics).

What part of I don’t want to use Rust which I have stated many times do you fail to assimilate?

So you did not disprove anything w.r.t. to my stated context.

The discussion started with you claiming that Rust’s ability to manage resource lifetimes was a big win for the borrow checker. And I followed by saying I had been thinking about I don’t want to implicitly conflate resource lifetimes with resource handle access lifetimes (which @keean is not the same as saying I want unchecked explicit finalization instead of RAII, come on keep orthogonal points orthogonal and not conflate them). And one of the reasons is because I don’t want to be forced to use Rust’s semantics and borrow checker (given that ARC is incompatible with my GC idea for ALPs). And the other reason is I think perhaps being implicit is obfuscating and can at least lead to unreadable code, which was exemplified in your subsequent example wherein @sighoya did not immediately realize that the function could have thrown an exception leading to a leak without implicit RAII. In other words implicit is not always conveying all the cases of implicit semantics to the programmer. Now whether that is a good or bad thing is a separate debate. But please come down off your “I disproved something” high horse.

[shelby3]

@shelby3 - Can you please just write a code example that actually truly genuinely shows your point that resource starvation is a likely outcome?

1. That was not my only point against being implicit. I also cited unreadable code and incompatibility with my ALP GC given I don’t want to add the complexity of Rust’s lifetimes model.

2. Never did I claim “likely outcome.” The point of such an example is to show how encapsulation can cause the resource release to be conflated with release of other things (perhaps another resource) that isn’t to be released at the same time. The last use of the reference to a data structure which somewhere in the object graph references a resource handle (and especially opaquely if you allow cyclic references) doesn’t correspond necessarily to end of the last access of the resource handle or the resource. Of course there will probably always be ways to refactor code to separate the concerns, but my point is a conceptual one in that the implicitness causes the programmer to not put much effort into keeping those concerns separate and thus can inadvertently land in a situation where resources get starved and then he has to go hunting for why and then mangle his code logic to unconflate reference access lifetimes from resource lifetimes which are not semantically isomorphic. And the reader of the code may have been none the wiser also because of all the implicit action such as when @keean deletes something from a Map then buried deep in the data structure (not in his example but hypothetically) of what was removed may be a reference to a resource. So maybe nobody was even thinking about when that resource actually gets released. It just happens opaquely and automagically.

Like @keean I have spent a lot of time with RAII-based languages (C++ and Rust) and a lot of time with GC-based languages with using (mainly Scala).

Remember you have pointed out that using is insufficient because it can’t check on resource lifetimes that escape the stack scope. So you are comparing RAII to an inferior paradigm. I was hoping to find someway to have checking without going all the way to Rust’s model, but I am now thinking it may not be possible. So I may have to paradigm-shift my entire approach to this thorny issue.

When it comes to correctness in handling non-memory resources, empirically my observation is that it is far easier to do it in C++ and Rust. It's a constant problem with my Scala programs. It's basically never a problem in my Rust programs. (In C++ I had so many other problems that I can't recall how bad it was, just that various other things were worse.)

This makes sense to me. But it is not a rebuttal to my desire to not want to be forced to use something as tedious (and ostensibly fundamentally unsound type system) as Rust. And punting to ARC will also not meet my ALP zero cost GC goals. Also ARC can not collect cyclic references! I think @keean may have forgotten that, otherwise maybe he would not be pitching ARC as a slam dunk.

EDIT: also apparently Rust can’t do unrestrained cyclic references (although some special casing of data structures with cyclic references apparently can be accommodated):

https://stackoverflow.com/questions/20698384/what-lifetimes-do-i-use-to-create-rust-structs-that-reference-each-other-cyclica/20704252#20704252

https://www.reddit.com/r/rust/comments/6rzim3/can_arenas_be_used_for_cyclic_references_without/

When it comes to explicit vs. implicit for things like this, I don't have a language that catches every case and insists that I be explicit about it, but I do have experience with parallel constructs (e.g. be-explicit-with-all-types, be-explicit-with-all-exceptions-thrown, be-explicit-with-all-returns) and in every case being explicit is a drawback. The reason is that attention is finite, and it wastes your attention on the obvious stuff that is working right every time (or which you can't even tell is working right because you're not a computer and can't trace every logical path in milliseconds like it can).

I was not proposing to be explicit about every catch and finally so the comparison is incorrect, as you admit below...

Even saying raii val handle = new Handle is an extra hassle, an extra bit of type information that actually isn't useful in the vast majority of cases.

Whether it is useful or not is an orthogonal debate, but at least it invalidates your comparison above.

In addition to the extensive experience that this is the better way to do things (at least for someone with my cognitive capacities), there is also the critical invariant that @keean keeps mentioning over and over again, which you never have adequately and directly responded to, which is that RAII can prevent use-after-free (and use-before-initialization) errors.

I have addressed it. You must have forgotten the entire discussion[my ranting] about bifurcation of high-level and low-level. Need I elaborate? I presume you understand that a high enough level language does not have use-after-free (and use-before-initialization) errors.

So, write a code example that shows what you mean! So far you've failed every time.

If you continue to make false allegations like that, then the exchanges between us are going to become more combative. You take something completely out-of-context and then make claims which thusly don’t apply.

Everything is trivially fixable or isn't even the problem you claim it is.

Nope. With that claim you implicitly presume or claim Rust is trivial. And by transitive implication you claim that ARC can handle cyclic references because my context has clearly been that Rust is not trivial, so the only alternative to Rust’s lifetimes currently concretely known to work for RAII is ARC.

EDIT: and apparently Rust can’t implement unfettered cyclic references well either. So your hubris has been deflated. Hey I was in an amicable mood of discussion. And then you and @keean started to attack me with an attitude of hubris with confident, false claims about how wrong I am, how you disproved me, how I don’t understand terminology, how I don’t understand the benefits (and tradeoffs) of RAII, ARC, Rust, etc... Tsk, tsk.

There’s always some tradeoffs in every paradigm choice. We should not declare that all the possible quadrants of the design space have been enumerated because it is difficult to prove a negative.

[shelby3]

Replying to myself:

The last use of the reference to a data structure which somewhere in the object graph references a resource handle (and especially opaquely if you allow cyclic references) doesn’t correspond necessarily to end of the last access of the resource handle or the resource.

So it should be easy to make an example that shows how this could lead to resource starvation which would not be detected by RAII (neither Rust nor ARC).

Just store the resource handle in a data structure and stop accessing it meaning you are finished with the resource handle, but continue to hold on to the reference to the data structure and access other members of that object. So neither Rust nor ARC will detect that the semantic resource life has ended before that of the lifetime of the data structure which contains it.

The only way around that is refactor the code to attempt remove the encapsulation (which has potentially other deleterious implications) or explicitly delete a strong reference (which @keean points out it is unsafe because the resource handle object would still be accessible).

I mentioned this scenario in my prior example (such as quoted below) and @NodixBlockchain also mentioned it.

So @keean may try to argue that the above examples are contrived and not representative other ways to refactor or thinking about implementing certain semantics. Yet my generative essence point (and this is why I say it is meta-level conceptualization and the coherent understanding can not be entirely conveyed in code) is that realize that Record may have more than one functionality and other references to it may come to exist dynamically due to other functionalities the data structure serves which may be entirely orthogonal to its functionality of also holding a weak resource handle for some functionality. You may myopically attempt to claim that you can always refactor code to make all concerns orthogonal but we simply know that’s impossible in programming, or at least where it is possible it can cause the programmer to so obfuscate the essential logic of the code that the code becomes unreadable, unmaintainable, and thus more likely to contain or accumulate (over time) bugs.

Essentially what I am saying is that the optimal resource lifetime is a separate concern from ARC lifetimes and by tying them together you discourage the programmer from modeling the optimal in his reasoning. Or you force the programmer to refactor code to maintain semantic isomorphism such that the code obfuscates the essential intent of the code.

@Ichoran what have you disproved? :facepalm:

EDIT: even just general discussions about memory leaks applies to why ARC can leak resource lifetimes if we conflate them with ARC (or for that matter RAII as implemented in Rust, because the lifetimes checker can’t resolve the last paragraph below which was my point all along):

https://www.lucidchart.com/techblog/2017/10/30/the-dangers-of-garbage-collected-languages/

Memory can still leak

The novice programmer may be misled into believing garbage collection prevents all memory leaks, but this is not the case. Although garbage collection prevents many types of memory leaks, it doesn’t prevent all of them.

In automatic reference counting systems, such as Perl or Objective-C, memory is leaked whenever there are cyclical references, since the reference count is never decremented to zero. The solution in these systems is to break the cycle by specifying that at least one of the references is a “weak” reference, which doesn’t prevent the object from getting garbage collected.

But even in languages with mark-and-sweep garbage collection where cyclical references are correctly garbage collected, such as Java, Javascript, and Ruby, there are still several ways to leak memory. These leaks occur when objects are still reachable from live objects but will never be used again. There are a number of ways this could happen.

[shelby3]

So now perhaps you can understand my perspective.

Given that I can’t have RAII in Task because:

1. I don’t want the tsuris of Rust’s lifetimes which includes even inability to accommodate unfettered cyclic references in addition to numerous other points that have been, including that we have to punt to ARC in Rust when our dynamic needs outstrip what static lifetimes can model and @keean claiming out he can’t implement certain algorithms without unsafe code. @keean would slices not help?

2. I don’t want the loss of performance of ARC (as compared to my novel GC in my ALP design conceptualization) nor do I want to leak cyclic references as ARC does.

And given that RAII has semantic vulnerabilities to resource starvation and implicit hiding of resource lifetimes as I have explained, then any solution I come up with could also have some vulnerabilities to resource starvation and not be at a disadvantage to RAII in every case.

(to be continued after I sleep with an explanation of my paradigm-shift idea, which isn’t that far from the original idea I was attempting to explain since yesterday)

[keean]

@Ichoran To summarise my comments on how we could do better than Rust, I think lifetime erasure is a problem in Rust. I propose a system that uses reference counting as its semantics, and the type system is then used to provide static guarantees over this. Where static behaviour can be proven, then the RC will be optimised out. The reason for ARC rather than Mark-Sweep memory management is then the semantics with destructors will be the same whether the compiler statically manages the memory or uses runtime/dynamic memory management. This allows using RAII consistently. The first pervasive simplification would be when there is only one reference (effectively an owning reference). In effect we have three kinds of reference:

Owning references: Unique Reference (like C++ unique pointer) Shared Reference (like C++ shared reference, needs ARC)

Not owning references (better name?): Strong Reference (where the compiler can statically prove the reference lifetime is shorter than the resource lifetime) Weak Reference (in all other cases where the required proof for a strong reference cannot be made).

Notes:

• A unique reference is just a special case of a shared reference, where the RC = 1
• We don't have to do any optimisation, we could just use weak references everywhere, which would be safe. So we can start with simple escape analysis for strong references.

So the idea is, unlike Rust we start with memory safe runtime semantics, and we optimise performance when we can make compile time proofs. So unlike Rust we don't need heuristic rules to keep the expressive power in the language, and static optimisations can be restricted to only those places we can actually prove the optimisation is safe.

This also avoids two things I don't like about Rust. There is no safe/unsafe code distinction, just performant and less performant code, and there is no source code distinction between ARC and non-arc code. This really helps with generics because we don't need to provide two different versions of generic functions to cope with ARC and non-ARC data.

@shelby3 If you want to use Mark Sweep style memory management, you would have to avoid static destructors to allow the compiler to optimise between runtime/dynamic memory management and static memory management with no change in the semantics. So the alternative architecture would be Mark Sweep with explicit destructor calls for non-memory resources.

My hypothesis, which could be wrong, is that enough code will optimise with static memory management that the performance difference between ARC and MS will not be noticeable. I think some people will prefer the RAII approach, and if we can negate the performance penalty of ARC with static management optimisations then that will be an interesting language.

I think both of these languages (RAII+ARC and ExplicitDestructor+MS) will be a lot simpler than Rust because we can hide all the lifetimes from the programmer, because we have safe runtime semantics with RC or MS, and then static lifetime management is an optimisation. We can implement increasingly sophisticated lifetime analysers without changing the semantics of the language, something Rust cannot do because it differentiates between dynamically managed resources (ARC) and statically managed resources in the language.

[Ichoran]

@shelby3 - You can use std::sync::Weak to make cyclic references safely and with no lifetimes (use move semantics). You can use only safe Rust, have no memory leaks, and good (but not zero-overhead) runtime. But it's a pain. You have to manually construct things without cycles by imposing an orientation (and also hang on to the root so it doesn't get collected).

The downside of RAII is basically the opposite of what you're saying. It's not that it has semantic vulnerabilities. It's amazing at avoiding vulnerabilities compared to everything else that anyone has come up with. However, it comes with some awkwardness in that you have to pay attention to when things are accessible and get rid of them as soon as they're not. This is a heavy burden compared to standard GCed memory management where things get lost whenever, and then eventually it's noticed at runtime and cleaned up.

If one doesn't mind the conceptual overhead of having both systems active at the same time, one could have the best of both worlds. A resource could either be another way to declare something, e.g. val x = 7; var x = 12; resource val x = 19 at which point RAII semantics would work for it. If put into a struct, the struct would have to be a resource, etc.. Or it could be a smart pointer type, Resource<X>. And then everything that wasn't a resource wouldn't have to care about when it appeared and disappeared; it would be specific to that smart pointer type (or declaration).

Alternatively, one can have general-purpose resource-collecting capability in addition to memory. You'd have to provide some handles to detect when resources are growing short, and adjust the GC's mark-and-sweep algorithm to recognize the other kinds of resources so they could be cleaned up when they grow short without having to do a full collection (though generally the detection is the tough part anyway). Then every resource would act like GC--you'd never really know quite when they'd be cleaned up, but whenever you started to get tight they'd be freed. Sometimes this is good enough. Sometimes it's risky. (E.g. not closing file handles can increase the danger of data loss.)

Regarding how to make RAII fail:

Just store the resource handle in a data structure and stop accessing it meaning you are finished with the resource handle, but continue to hold on to the reference to the data structure and access other members of that object.

Yes, absolutely. If you hang on to a resource on purpose by sticking it in a struct then, sure, it's not going to get cleaned up because you might use it again.

Any resource can be held onto that way--actually finish using it, but don't say so, and require your computer to solve the equivalent of the halting problem in order to determine whether it'll actually get used again.

If you have users who are using stuff that is a resource without even knowing that it's a limited resource, and are making this kind of mistake a lot, then yeah, okay, I see that they might need some help. Having the type system yell at them when they create it to make sure they know what they're getting into is perhaps a plus. ("I opened a file? And I can't open every file at the same time, multiple times, if I want to? Golly gee, these computers aren't so flexible as everyone makes them out to be!")

If you have very slightly more canny users, they presumably won't do that. They'll learn what things use scarce resources, and use whatever the language provides for them to release them when they need to be.

[Ichoran]

@keean - The cognitive overhead in Rust of avoiding Arc cycles is not entirely negligible--data structures where you used to not have to care end up being a mix of Arc and Weak. Do you have a plan for how to avoid that?

[keean]

@Ichoran

The cognitive overhead in Rust of avoiding Arc cycles is not entirely negligible

Yes, owning pointers must form a Directed Acyclic Graph, enforced by the type checker.

[shelby3]

@keean any feedback yet from Github?

If we loose that entire issues thread #35, that is somewhat catastrophic to my work on the new PL. There were so many important comments posts that I had linked to from my grammar file which contained various explanations of various issues in the PL design space.

I know you have that backup you made 2 years ago. At the moment I still have a copy of that #35 issues thread still open on my web browser. So in theory I could start copying (at least my) hundreds of posts out the thread (or at least the ones which were added or changed since your backup), but that would be a significant manual undertaking, unless perhaps I figured out how to write some scraper in JavaScript or a browser plugin.

Does Github have a paid support level? Should we pay to get the service we need? How much would it cost? I might be willing to chip in to get that issues thread restored.

Also it may be the case that if they restore, they do it from a nightly backup so perhaps we may still lose some of the last comment posts. So the sooner I could know, the more chance that the copy open in my browser window will still be there, so I can recover the last comment posts as may be necessary. So what I am saying is can you push harder on Github support for feedback sooner?

[keean]

@shelby3 nothing heard back yet. Its probably be sensible to make sure you don't lose what you have, but hold off posting up yet.

[shelby3]

@keean

I haven’t read the context yet, but I actually thought this point might be raised so I was thinking about this already just before I drifted off to sleep...

@Ichoran

The cognitive overhead in Rust of avoiding Arc cycles is not entirely negligible

Yes, owning pointers must form a Directed Acyclic Graph, enforced by the type checker.

That Rust essentially prevents one from creating willy-nilly cyclic references, isn’t necessarily a good thing. It’s a limitation which may impact on degrees-of-freedom. I have not fully explored what that limitation means in practice, but I have presume that being able to not restrict cyclic references is preferred in a general purpose PL, if it doesn’t incur some unacceptable tradeoff such as the level of performance priority required for the use case (which for example in a high-level language the performance priority is somewhat lowered relative to an increase in the need for degrees-of-freedom, flexibility/ease-of-expression, unobtrusive safety guarantees, etc)?

And ARC doesn’t prevent cyclic references yet can’t collect them. And I’m aware of partial tracing algorithms that attempt to cleanup cyclic references, but these don’t guarantee to catch them all without perhaps the extreme of essentially trashing the overall performance.

Also for any reentrant or multithreaded code, then ARC requires a multithread synchronization primitive on each modification of the reference count. There are other issues with performance which have some workarounds with some designs but various tradeoffs:

https://en.wikipedia.org/wiki/Reference_counting#Dealing_with_inefficiency_of_updates https://news.ycombinator.com/item?id=10151176 https://www.quora.com/Why-doesnt-Apple-Swift-adopt-the-memory-management-method-of-garbage-collection-like-Java-uses https://softwareengineering.stackexchange.com/questions/285333/how-does-garbage-collection-compare-to-reference-counting https://www.quora.com/Why-dont-other-modern-programming-languages-support-Automatic-Reference-Counting-like-Objective-C-Swift-but-use-garbage-collection-instead

So this seems to point towards Rust as the only solution which is more performant while still admitting some cases of (but AFAIK not truly unrestrained) cyclic references, yet I would prefer to find a solution which is nearly as performant and zero cost abstraction as Rust (including lowest memory footprint), without any of the complexity of the lifetime tracking (annotations, limitations, unsoundness, etc) and which can fully integrate with cyclic references as tracing GC does.

I think I may have found another quandrant in the design space. There will be tradeoffs of course — there’s always some caveat. AFAICS, our job is to identify the tradeoffs and make engineering choices.

[shelby3]

@keean

@shelby3 nothing heard back yet. Its probably be sensible to make sure you don't lose what you have, but hold off posting up yet.

In the meantime could you ask the community what Github’s typical capabilities and reaction is to such a support request and what is the best way to go about seeking any remedies which may be available?

https://github.community/

Are you sure the history is not available via an API? Surely the issues threads are Git versioned, including deletes? Perhaps the community may have a solution which for us which doesn’t require action from Github Support, whose webpage for opening a support ticket states they’re currently understaffed due to “COVID-19”[^{_{China’s psyops revenge for Hong Kong’s intransigence and the U.S.A. turning against their unfettered mercantilism[1], among other Machiavellian goals of our overlords}}].

[1] _{Skip to p.265 of Bolton’s book to read the chapter about how China manipulated Trump. Keep in mind that Bolton is a war hawk and his book is written to inflame conservatives. Nevertheless I think it’s possibly also a valid indictment of China’s internal politics.}

[keean]

@shelby3 When I searced for issues, it seems they normally restore deleted stuff, next day for some people, but they say they are responding more slowly due to the current situation.

[shelby3]

@keean

@shelby3 When I searced for issues, it seems they normally restore deleted stuff, next day for some people, but they say they are responding more slowly due to the current situation.

That’s a major relief. Thank you. I hope so.

[NodixBlockchain]

@keean

@shelby3 When I searced for issues, it seems they normally restore deleted stuff, next day for some people, but they say they are responding more slowly due to the current situation.

That’s a major relief. Thank you. I hope so.

i have the posts in mail from 03 jan 2018

[NodixBlockchain]

@Ichoran To summarise my comments on how we could do better than Rust, I think lifetime erasure is a problem in Rust. I propose a system that uses reference counting as its semantics, and the type system is then used to provide static guarantees over this.

It's in the idea of what i'm doing with the runtime, can switch from rc to ms, and still keep the rc semantic, that can help tracking subtle memory bugs, and make the required memory pattern more obvious.

[shelby3]

@Ichoran

@shelby3 - You can use std::sync::Weak to make cyclic references safely and with no lifetimes (use move semantics). You can use only safe Rust, have no memory leaks, and good (but not zero-overhead) runtime. But it's a pain. You have to manually construct things without cycles by imposing an orientation (and also hang on to the root so it doesn't get collected).

The strong and weak references paradigm is a pita. Although perhaps it should be an option, I’d hope (at least for a high-level, easy-to-use PL) not to choose a paradigm where it’s the only option, because it’s another deviation from the ideal degrees-of-freedom, being sort of another What Color Is Your Function bifurcation — which also describes at the generative essence Rust’s alternative to not using weak references ostensibly means an inability to have any cyclic references.

The downside of RAII is basically the opposite of what you're saying.

Again AFAICS you’re (perhaps unwittingly due to inadvertent choice of wording and not with tendentious nor combative intent[1]) making false allegations by insinuating that I didn’t recognize the downside of the tradeoffs (e.g. “heavy burden”) incurred to implement RAII and/or that I claimed that RAII has the downside of not improving upon unchecked lifetimes semantics, which I did not claim. Let me quote myself again:

But the points I made against implicit RAII were not that it can’t be made to work and were not a claim that it isn’t convenient and doesn’t prevent bugs compared to an unchecked manual cleanup (which I never advocated completely unchecked manual cleanup!), which just goes to show you have not even bothered to really understand what I wrote.

My points are about the obfuscation in code it creates and how that can lead to suboptimal outcomes and unreadable code in some cases. Surely you can fix bugs and mangle your code to make it all work, but you may make it even more unreadable. My point was an attempt think about how to make it even better […]

[…]

I think your criticisms of RAII amount to a personal preference, and you don't seem to appreciate the real word benefits that I, and some others have been trying to explain to you, and are rooted in practical experience not speculation.

Have I ever stated I don’t appreciate the benefits of RAII as compared to manual unchecked explicit release of resources? No! If you think I did, you simply failed to read carefully.

---

It's not that it has semantic vulnerabilities.

Yet it does have semantic vulnerabilities which you finally admit below.

It's amazing at avoiding vulnerabilities compared to everything else that anyone has come up with.

I never claimed otherwise, although I wouldn’t go quite so far as the hyperbole[1] “amazing” because it doesn’t prevent naive programmers from “live leaking” (i.e. not just forgetting to deallocate but deallocating too late) and it does lead to opaque implicitness — remember I am trying to design a PL that could possibly be popular which means as I have documented hordes of naive, young, male programmers with less then 5 years of experience (they significantly outnumber the experts). Does wanting to improve upon RAII have to mean (conflate) in your mind that I think the implicit cleanup of RAII is the worst thing ever invented?

I attempt to improve upon RAII because up until now the only ways to achieve it have been the all-or-nothing tsuris of Rust or the limitations/downsides of ARC. Meaning that although I think it was an improvement in some respects over the paradigm the programmer has been offered for example with a MS (aka tracing) GC, that the tradeoffs of using it are countervailed to some extent by significant tradeoffs in other facets of the PLs that offer RAII as @keean, you and I have discussed. And especially so the tradeoffs when we consider the aforementioned demographic I might be targeting, and in general the point I have made about a bifurcation between high-level and low-level PL taken in the context of my desire to focus first (for now) on the high-level and making it as fun and easy to work with as possible by not conflating it with low-level concerns. Note there are many orthogonal statements above, so (to all readers) please don’t conflate them into one inkblot. Give me some benefit of understanding please. Our discussions should be about trying to enumerate the ranges of the PL design space and raising all of our understandings and elucidations of the various possibilities which have already and have not yet been explored.

However, it comes with some awkwardness in that you have to pay attention to when things are accessible and get rid of them as soon as they're not.

Okay we are making progress towards some consensus of understanding and claims.

This is a heavy burden compared to standard GCed memory management where things get lost whenever, and then eventually it's noticed at runtime and cleaned up.

Agreed as I understand that Rust forces the programmer to prove certain lifetime invariants. I just want to add and note that I had already mentioned that Rust can also allow a “live resource leak”. My point being that the “heavy burden” paid for RAII as implemented in Rust does not resolve all risk of a “live resource leak”.

If one doesn't mind the conceptual overhead of having both systems active at the same time, one could have the best of both worlds. A resource could either be another way to declare something, e.g. val x = 7; var x = 12; resource val x = 19 at which point RAII semantics would work for it.

AFAICS, merging RAII with a MS (aka tracing) style GC’ed language would incur the same lifetime proving “heavy burden” as Rust (because ARC can’t be merged without creating a What Color Is Your Function bifurcation) unless as @keean’s post today is leading towards (@keean you took some of my design idea from my mind while I was sleeping but you’re not all the way there) the design instead becomes it’s only a best effort and not a guarantee.

Alternatively, one can have general-purpose resource-collecting capability in addition to memory. You'd have to provide some handles to detect when resources are growing short, and adjust the GC's mark-and-sweep algorithm to recognize the other kinds of resources so they could be cleaned up when they grow short without having to do a full collection (though generally the detection is the tough part anyway). Then every resource would act like GC--you'd never really know quite when they'd be cleaned up, but whenever you started to get tight they'd be freed. Sometimes this is good enough. Sometimes it's risky. (E.g. not closing file handles can increase the danger of data loss.)

I guess we could teach a GC to have prioritization for cleaning up certain resources when they become starved but this is not research I am aware of and it sounds to me that it would have many pitfalls including throttling throughput. I am not contemplating this design. My design idea essentially parallels what @keean wrote today but combined with my new ALP-style GC which requires no MS (no tracing).

Regarding how to make RAII fail:

Just store the resource handle in a data structure and stop accessing it meaning you are finished with the resource handle, but continue to hold on to the reference to the data structure and access other members of that object.

Yes, absolutely. If you hang on to a resource on purpose by sticking it in a struct then, sure, it's not going to get cleaned up because you might use it again.

Thank you. Finally we presumably have some level of consensus that I am not completely batshit crazy, don’t “you've failed every time”[sic] and was not writing complete nonsense.

Any resource can be held onto that way--actually finish using it, but don't say so, and require your computer to solve the equivalent of the halting problem in order to determine whether it'll actually get used again.

Which BTW is the generative essence of why Rust’s lifetime checker has no prayer of ever being able to analyze every case where something leaks semantically or something is safe but Rust thinks it is unsafe.

And remember I wrote in my ranting about Rust that Rust can’t catch all semantic errors. (Not implying you disagreed with that claim).

If you have users who are using stuff that is a resource without even knowing that it's a limited resource, and are making this kind of mistake a lot, then yeah, okay, I see that they might need some help. Having the type system yell at them when they create it to make sure they know what they're getting into is perhaps a plus. ("I opened a file? And I can't open every file at the same time, multiple times, if I want to? Golly gee, these computers aren't so flexible as everyone makes them out to be!")

I believe you at least slightly mischaracterize the vulnerability. The convenience of being implicit (just declare a destructor and fuhgeddaboudit) can lead to not paying careful attention.

@keean has for example documented how not-so-naive programmers leak in JavaScript with implicit closures. Implicitness encourages not thinking.

If you have very slightly more canny users, they presumably won't do that. They'll learn what things use scarce resources, and use whatever the language provides for them to release them when they need to be.

They will make less errors, but they can still be snagged. Don’t me tell you never have because I will not believe you.

[1] _{It’s understandable that possibly my rants against Rust have presumably caused (perhaps subconsciously) you to push-back with more vigor than you would had I not expressed what you probably perceive to be unnecessarily too aggressive, flippant, one-sided, uncivil, discourteous, incendiary, emotionally charged, unprofessional, community-destroying, self-destructive, etc..}

[shelby3]

@keean

I propose a system that uses reference counting as its semantics, and the type system is then used to provide static guarantees over this. Where static behaviour can be proven, then the RC will be optimised out. The reason for ARC rather than Mark-Sweep memory management is then the semantics with destructors will be the same whether the compiler statically manages the memory or uses runtime/dynamic memory management. This allows using RAII consistently.

[…]

So the idea is, unlike Rust we start with memory [i.e. resource leak and use-after-free] safe runtime semantics, and we optimise performance when we can make compile time proofs. So unlike Rust we don't need heuristic rules to keep the expressive power in the language, and static optimisations can be restricted to only those places we can actually prove the optimisation is safe.

This also avoids two things I don't like about Rust. There is no safe/unsafe code distinction, just performant and less performant code, […]

@keean I like that you were trying to paradigm-shift in a way that somewhat analogous to the paradigm-shift I’m contemplating, but unfortunately there’s a flaw in your design idea.

[…] and there is no source code distinction between ARC and non-arc code. This really helps with generics because we don't need to provide two different versions of generic functions to cope with ARC and non-ARC data.

The flaw is that contrary to your claim of avoiding a What Color Is Your Function-like bifurcation, because there’s still such a bifurcation in your design.

This will still be a bifurcation w.r.t. to generic functions that can’t be monomorphised statically because non-ARC and ARC data can’t fully interopt.[c.f. correction] You can assign non-ARC data to a reference in an ARC data structure but you can’t allow assignment between incompatible non-ARC and ARC because the write barrier for ARC data operates on the refcount.

You may perhaps consider that flaw to be a worthwhile tradeoff but frankly given that ARC leaks cyclic references then I think it’s a non-starter for me to even consider.

And I don’t think your idea will be as performant as my paradigm-shift which I will attempt to explain today.

@shelby3 If you want to use Mark Sweep style memory management, you would have to avoid static destructors to allow the compiler to optimise between runtime/dynamic memory management and static memory management with no change in the semantics. So the alternative architecture would be Mark Sweep with explicit destructor calls for non-memory resources.

My ALP design (c.f. also) doesn’t employ MS nor tracing. It’s a bump pointer heap (i.e. nearly as efficient to allocate as static stack frames) which is released in entirety with a single assignment to reset the bump pointer in between the processing of each incoming message of the ALP.

Although you’re correct that in lieu of the guarantee of RAII via ARC (even Rust’s lifetime model requires ARC because it can’t model everything statically!), some explicitness is required, it will not preclude some of the static analysis you were mentioning for your paradigm-shift idea — it just won’t be a 100% RAII guarantee. But here’s the kicker which you were missing in your presumptions about what I am formulating: the guaranteed fallback in my ALP idea is still a 100% check and that typically occurs reasonably soon as the ALP returns to process the next message in the queue (although this will require overhead in addition to the single-assignment for resetting the heap, but this will only occur for those not explicitly tagged as RAII, which of course the compiler statically checks). If the programmer is doing something which can cause a long pause, they out to make sure that either the statically proven RAII was implicitly carried out (i.e. destructor called, which is the reason for some terse explicit keyword) or they should manually finalize. EDIT: There appears to be another place in the design space.

My hypothesis, which could be wrong, is that enough code will optimise with static memory management that the performance difference between ARC and MS will not be noticeable.

Your ARC will still be less performant than my bump pointer heap with single assignment reset (no mark, no sweep, no tracing, nothing aka nearly “zero cost abstraction”). Mine should even better approach Rust’s performance than Go, and without the tsuris “heavy burden”.

I think some people will prefer the RAII approach, and if we can negate the performance penalty of ARC with static management optimisations then that will be an interesting language.

Well if not for the aforementioned bifurcation flaw I think you might be onto another interesting possibility in the design space. But that flaw taken together with leaking of cyclic references significantly dampens my interest. Also the performance would not be as good as what I’m contemplating. And also I have explained why I think implicit RAII is not an unequivocal guarantee of timely release of resources nor an inviolable guarantee against resource leaks. Also I think RAII encourages programmers to not optimize their resource lifetimes as I wrote, “The convenience of being implicit (just declare a destructor and fuhgeddaboudit) can lead to not paying careful attention.”

I think both of these languages (RAII+ARC and ExplicitDestructor+MS) will be a lot simpler than Rust because we can hide all the lifetimes from the programmer,

I agree except for the bifurcation flaw in your idea. I think my idea is superior although the guarantee on timely cleanup is different (but arguably better!)

because we have safe runtime semantics with RC or MS, and then static lifetime management is an optimisation.

Agreed.

We can implement increasingly sophisticated lifetime analysers without changing the semantics of the language, something Rust cannot do because it differentiates between dynamically managed resources (ARC) and statically managed resources in the language.

Unfortunately your idea must also per the flaw I mentioned. But AFAICS my idea does not!

[shelby3]

@NodixBlockchain

@keean

@shelby3 When I searced for issues, it seems they normally restore deleted stuff, next day for some people, but they say they are responding more slowly due to the current situation.

That’s a major relief. Thank you. I hope so.

i have the posts in mail from 03 jan 2018

Does it have the link for each post? Because I refer to the posts by their URLs, so finding which posts I cited would be implausible without the links.

[keean]

@shelby3

This will still be a bifurcation w.r.t. to generic functions that can’t be monomorphised statically

The source code of the function will be the same whether it can or cannot be monomorphised statically, so there will be no bifurcation of source. In some cases where we can prove static management of the resource is sufficient we will emit a different function, compared to those where the optimisation cannot be made. This second part is trivially obvious, because if we did not emit different code depending on the use case, we would not be optimising anything, and instead we would just have a language with ARC memory management.

[shelby3]

This will still be a bifurcation w.r.t. to generic functions that can’t be monomorphised statically

The source code of the function will be the same whether it can or cannot be monomorphised statically, so there will be no bifurcation of source. In some cases where we can prove static management of the resource is sufficient we will emit a different function, compared to those where the optimisation cannot be made.

You will not, but you will.

Readers @keean agreed with me.

This second part is trivially obvious, because if we did not emit different code depending on the use case, we would not be optimising anything, and instead we would just have a language with ARC memory management.

So all functions will be bifurcated. Which is what I said.[@keean is correct but with a potentially significant limitation on what can be optimized]

[NodixBlockchain]

@NodixBlockchain

@keean

@shelby3 When I searced for issues, it seems they normally restore deleted stuff, next day for some people, but they say they are responding more slowly due to the current situation.

That’s a major relief. Thank you. I hope so.

i have the posts in mail from 03 jan 2018

Does it have the link for each post? Because I refer to the posts by their URLs, so finding which posts I cited would be implausible without the links.

The post contain the (now broken) links, and there is the 'view it on github' link in the footer that contain the id, i guess it can be reconstructed but not easy.

[keean]

@shelby3

So all functions will be bifurcated. Which is what I said.

No, you will only need one generic definition of an algorithm like 'sort' to sort all ARC and non-ARC Data

[shelby3]

@shelby3

So all functions will be bifurcated. Which is what I said.

No, you will only need one generic definition of an algorithm like 'sort' to sort all ARC and non-ARC Data

@keean what am I missing? An assignment involving ARC requires a write barrier to update the refcount. Non-ARC will not have that write barrier. So are you saying your optimized non-ARC will also have a write barrier which will check the object to see there is no refcount? If so, then not entirely optimized? It seems this would fix the bifurcation but at the cost of some of the improved performance you were aiming for.

EDIT: are you saying that the programmer’s code for generic functions is not bifurcated, only that the compiler creates optimized variants of these generic functions for the non-ARC case? This must be what you are thinking, but it is still flawed. If so, then my point is there are two variants of the function, so the programmer can not call the generic function with both ARC and non-ARC intermixed. This a bifurcation.

EDIT#2: The only exceptions would be where you could analyze statically that there would be no intermixing, but in general this is undecidable (e.g. runtime input values to functions which dictate conditional code paths aren’t known at compile-time) due to the Halting problem thus there would be some bifurcation.

Note you could optimize away ARC in only those cases which don’t call functions which can’t be statically analyzed to confirm no intermixing violations. Thus by definition there would be no intermixing and thus no bifurcation. Perhaps this is what you meant.

[shelby3]

I want to finish the details for my paradigm-shit idea augmenting what wrote in my prior longish response to @keean.

So I have the analogous idea as @keean to do static analysis to determine which resource lifetimes (other than memory resources) can be implicitly statically collected as they would be for RAII. Since this static analysis is limited without lifetime annotations and depending on the program code being analyzed, not every case will succeed in being statically determined.

In @keean’s design those that fail to be statically determined default to ARC and thus are still RAII guaranteed, i.e. @keean is only attempting to improve performance of RAII because ARC is slower than non-ARC. Whereas, in my idea there’s no benefit to RAII collection of memory resources because (the memory won’t be recovered any sooner and) my bump-pointer-heap-with-assignment-reset (BPHWAR) idea is going to collect all the memory automatically with near zero-cost between each message which is processed from the actor-like parition’s (ALP) message queue. Thus my design goal for the static analysis isn’t to improve execution speed (although perhaps optionally allocating on stack frames instead of the bumper-pointer heap might be more performant, or at least allocating on the bumper-pointer heap in chunks instead of piece-meal might benefit speed), instead only to improve the determinism and optimization of the cleanup of resource lifetimes (again for resources other than memory). Because the default in my design (in the cases in which static analysis is not determined by the compiler, and the finalizer is not called manually by the programmer) is to automatically cleanup/release/deallocate all resources (including memory) between the aforementioned processing of the message queue.

So whereas @keean’s idea doesn’t require explicitly declaring which object instances will be RAII and which will not (because by employing ARC for those not statically determined to be non-ARC in his design still always guarantees RAII), my idea requires the programmer explicitly declare which object instances will be RAII so that the compiler can complain if they can’t be statically determined. Thus the programmer will be aware which instances are eligible for manual invocation of the finalizer. The programmer need not necessarily manually invoke the finalizer because the cleanup will happen normally soon enough in between the aforementioned processing of the message queue. But the programmer needs to be aware in the case where there will be some significant delay, so that appropriate manual finalization can be explicitly placed in the program code in that case so as to optimize the outlying scenarios.

Since the said explicit declaration via a keyword such as raii or auto is type tag of the object instance, this will be available to IDEs to indicate where the said instance propagates via assignment and from its function and into other functions. Thus the programmer via the IDE will have an indication which need to be manually optimized. Note most likely only move semantics (with implicit exclusive borrows) will be statically analyzed wherein there is only one strong reference to the statically analyzed object instance (and optionally with a plurality of weak references). When the strong reference is removed (i.e. dies) the object instance’s destructor is called and the data of the object can no longer be accessed by any weak reference. Whereas, when a finalizer is manually invoked the object instance’s data is still accessible so the programmer has to insure that the object is in a coherent state after finalization. Thus for the finalization case there’s no distinction between a strong and weak references and thus I guess weak references should only be allowed on those object instances which are not statically RAII determined.

The avoidance of bifurcation in my design idea is ostensibly somewhat more flexible than in @keean’s because there’s never any write barrier on assignment. The only statically compiled distinction is whether when a reference dies (and was moved and not borrowed, which thus exempts most functions), does the runtime call the object’s destructor. So whereas AFAICS @keean’s idea’s opportunities for optimization will be limited to those cases that don’t do intermixing on assignment, the static analysis in my idea will only have to check for intermixing that leads to (non-borrowed) reference death.

ALP’s Persistent State

The persistent object instances of each ALP (i.e. the ALP’s persistent state) are ARC, but this doesn’t cause a significant bifurcation because of some restricted semantics:

• They must be immutable. Thus not eligible for functions which would mutate the refcounts of internal references of an object instance.
• They’re not reference counted within the processing of a message from the queue (because all increments and decrements net to zero), which causes no issues because being immutable the refcounts of internal references can’t be changed.
• To replace the value of a persist object requires calling a special function which returns immediately but doesn’t initiate the replacement until the processing for the current message of the message queue has completed.

IOW and concisely, the ALP’s persistent state has immutable, atomic state machine transitions aka monadic semantics.

Keean’s ARC vs. My ALP’s BPHWAR

1. Mine doesn’t leak cyclic references. Keean’s does leak (unless programmer employs weak references) and Rust doesn’t allow them without introducing the complication of weak references. Unrestricted strong, cyclic references aren’t deleterious because by definition a MS (aka tracing) GC can collect them (as my non-MS, BPHWAR can also). Additionally my BPHWAR can even correctly collect, live cyclic references which would not have been collected by MS, preventing a semantic leak!
2. Mine requires an ALP paradigm.
3. Keean’s guarantees RAII cleanup. Mine can only RAII those statically determined (and explicitly tagged with a keyword on object instance construction, e.g. raii or auto). Mine guarantees cleanup between processing messages from the ALP’s message queue.
4. Keean’s is entirely implicit RAII. Mine encourages being explicit to obtain an RAII guarantee and otherwise being manually explicit with a finalizer where needed for resource lifetime optimization. Thus Keean’s encourages programmer to be lazy about thinking about resource lifetimes (“just declare a destructor and fuhgeddaboudit”) and I argued the guarantee can result in resource starvation leaks when the programmer doesn’t pay attention. Mine encourages the programmer to be vigilant about resource lifetime optimization, whilst also perhaps making most instances RAII or default implicit cleanup between processing messages from the ALP’s message queue. This is a difference in philosophical design of a PL.
5. Mine should be faster and faster than Go towards approaching Rust’s performance (to the extent a language which doesn’t conflate low-level capabilities with high-level can) without Rust’s tsuris and unsoundness. Mine should scale to massively multicore whereas Go’s monolithic GC for all green threads (aka goroutines) can’t.
6. Mine will prove data race safety which Go doesn’t, and without Rust’s tsuris nor even as abstruse as Pony.
7. Keean’s has a more restrictive limit on which instances of RAII can be statically determined as compared to mine.

[shelby3]

@NodixBlockchain

@NodixBlockchain

@keean

@shelby3 When I searced for issues, it seems they normally restore deleted stuff, next day for some people, but they say they are responding more slowly due to the current situation.

That’s a major relief. Thank you. I hope so.

i have the posts in mail from 03 jan 2018

Does it have the link for each post? Because I refer to the posts by their URLs, so finding which posts I cited would be implausible without the links.

The post contain the (now broken) links, and there is the 'view it on github' link in the footer that contain the id, i guess it can be reconstructed but not easy.

I only need the broken links so I can cross-reference the link in my grammar file to the content for each comment post. So does your email history of that #35 thread. Can you post an example of the “id” for the “view it on github” link so I can determine if what you have is sufficient?

I am becoming concerned again now that so much time as elapsed and the #35 thread has not yet been restored.

[keean]

@shelby3 some changes: RAII+ARC will not leak, because the compiler checks owning-references form a Directed Acyclic Graph. Unique References mean there can be only one owner - for example consider a tree where the parent owns the children by a unique reference - when the root is destroyed, all the children get automatically destroyed.

[keean]

@shelby3 they got back to me, but they only had the first post of the issue... I think they misunderstood what I wanted. I have pointed out that there were a lot of comments on that issue that we also would like restored if possible. I don't know if they will be able to do better than this.

[shelby3]

@shelby3 they got back to me, but they only had the first post of the issue... I think they misunderstood what I wanted. I have pointed out that there were a lot of comments on that issue that we also would like restored if possible. I don't know if they will be able to do better than this.

This is horrible. :sob:

[keean]

@shelby3 Sorry, it's my fault, totally didn't expect it to delete the whole thread and it not be recoverable.

[shelby3]

@shelby3 Sorry, it's my fault, totally didn't expect it to delete the whole thread and it not be recoverable.

It’s okay @keean. Thank you for trying to recover it. Everyone (of us) has made such an error in their lifetime. I will just have to manually recover it perhaps starting today. No other choice unless Github will restore it.

[shelby3]

@keean

@shelby3 some changes: RAII+ARC will not leak, because the compiler checks owning-references form a Directed Acyclic Graph. Unique References mean there can be only one owner - for example consider a tree where the parent owns the children by a unique reference - when the root is destroyed, all the children get automatically destroyed.

AFAICS, I already stated that in my summary of the differences between your idea and mine?

Indeed yours can avoid cyclic references with a single strong reference akin to std::unique_ptr and optionally multiple weak references. Disallowing cyclic references (by disallowing all strong shared references) is not the same as not leaking cyclic references. And disallowing std::shared_ptr has downsides such as being itself a bifurcation and adding significant complexity and failure modes that the programmer has to deal with.

https://en.cppreference.com/w/cpp/language/raii

As for the potential for the implicitness and conflation of ARC with resource lifetimes in RAII to leak resource lifetimes (especially among careless or novice programmers), I will not repeat my argument. I stand by it but I do not overemphasize it as it is a nuanced point for which we have no quantitative metrics to make concrete claims about the likelihood.

[shelby3]

@keean

@shelby3 Sorry, it's my fault, totally didn't expect it to delete the whole thread and it not be recoverable.

It’s okay @keean. Thank you for trying to recover it. Everyone (of us) has made such an error in their lifetime. I will just have to manually recover it perhaps starting today. No other choice unless Github will restore it.

I believe I have captured the HTML source of the entire deleted #35 thread:

https://drive.google.com/file/d/1euv36tcdVM_dSElMeHGt9qpvemrI5ot6/view?usp=sharing

Do you feel like writing a script that employs the Github API to reconstruct the thread?

Since you can’t post on another user’s behalf, I think the script would need to attribute within each reconstructed comment who the original author was. Also the original ID of the comment post should be included in the comment post, so that links to comments in the deleted thread can be discovered by searching the page on the comment id. Something like this:

[reconstructed] WD-40 (for reducing Rust with the next mainstream language) #35

Or does anyone have a better idea for how to reconstruct this information in a suitable format that meets the above objectives?

[NodixBlockchain]

@NodixBlockchain

@NodixBlockchain

@keean

@shelby3 When I searced for issues, it seems they normally restore deleted stuff, next day for some people, but they say they are responding more slowly due to the current situation.

That’s a major relief. Thank you. I hope so.

i have the posts in mail from 03 jan 2018

Does it have the link for each post? Because I refer to the posts by their URLs, so finding which posts I cited would be implausible without the links.

The post contain the (now broken) links, and there is the 'view it on github' link in the footer that contain the id, i guess it can be reconstructed but not easy.

I only need the broken links so I can cross-reference the link in my grammar file to the content for each comment post. So does your email history of that #35 thread. Can you post an example of the “id” for the “view it on github” link so I can determine if what you have is sufficient?

I am becoming concerned again now that so much time as elapsed and the #35 thread has not yet been restored.

```Reply to this email directly, 
```<a href=3D"https://github.com/keean/zenscript/issues/35#issuecomment-346119442">view it on GitHub</a>
```<div itemscope itemtype=3D"http://schema.org/EmailMessage">
```<div itemprop=3D"action" itemscope itemtype=3D"http://schema.org/ViewAction">
 ```<link itemprop=3D"url" href=3D"https://github.com/keean/zenscript/issues/35#issuecomment-346119442"></link>
```<meta itemprop=3D"name" content=3D"View Issue"></meta>
```</div>
```<meta itemprop=3D"description" content=3D"View this Issue on GitHub"></meta>
```</div>

There is also a script like this with metas 

```<script type=3D"application/json" data-scope=3D"inboxmarkup">
{"api_version":"1.0",
"publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},
"entity":
{"external_key":"github/keean/zenscript",
"title":"keean/zenscript",
"subtitle":"GitHub repository",
"main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png",
"avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png",
"action":{"name":"Open in GitHub","url":"https://github.com/keean/zenscript"}},
"updates":{"snippets":[{"icon":"PERSON","message":" xxx "}],"
action":{"name":"View Issue","url":"https://github.com/keean/zenscript/issues/35#issuecomment-346119442"}}}
</script>

[NodixBlockchain]

this is what i mannaged to pry out of my mail box (never thought it would be such a hassle to export mails :D ) Mails.zip

[shelby3]

@Ichoran

I am going to delete the reactionary, incendiary crap I wrote which was directed at you personally. It will still be in the edit history for anyone who wants to bother.

We are being psychologically abused[tortured] at this juncture in history and the following explanatory video doesn’t stake out any ideological position:

https://m.youtube.com/watch?v=GWk-ZpJdRFg (Making Sense of the Downward Spiral, Daniel Schmachtenberger)

A very smart friend of mine reminded me that psychosis is a disease of living in the past instead of creating the new — something that has been very challenging for me to do as I would therapeutically before I become so frustrated with my daily health battle. And now on top of that I have to deal with societalcide collapse, and I am living all alone (i.e. no social outlets, no human interaction, no human touch) for months as I just moved back in the U.S.A. after living in the Philippines non-stop since 2006. Even all the basketball courts (one of my favorite ways to blow off steam outdoors, be creative, and express youthfulness) are still cordoned off with police tape.

I express my dissatisfaction with Rust and in general with the clusterfuck of Node/JavaScript because I would prefer to be programming an application, not the mind-bending work of creating the PL I realized around 2008 I needed. I tried HaXe and then Scala. And still wasn’t satisfied. Scala was lacking for example the non-disjoint structural unions (which are now added to Scala 3), but I since realized I need green threads on the server and I want to use the same high-level PL on both server and client. You will note I often praise goroutines but lament that Go lacks data race safety. Go created an innovation I want. Pony exemplified that data race safety might not need to require a total program order on exclusive write ownership. Some of my angst against the Power Rangers who create Rust/Node/JS is amplified because I perceive them to be from the “aspirational class” of snotty socialists and social engineering merged into technology. You perceive that I am conflating technological discussion with social engineering concerns — well it is because they did! As I documented, it even started back near the turn of the century with Mozilla’s religious war against for example the site-supplied text. I am speaking out against doing that and want to keep our discussions on track of engineering and leave the social engineering crap out of our field.

Essentially what bothers me is I had appreciated so much your interaction. And I had even tried to intervene if I saw that @keean had not entirely characterized some of your points. And I tried to be fair and objective. But then when I express strong opinions that I want something different than the status quo of PLs, it seems you feel I am just a hot air balloon that likes to criticize the significant “accomplishments” of extant projects whilst I have accomplished “nothing” so to speak (although I think I have accomplished a significant amount of design thought already).

Please understand I am very frustrated with my slow progress. And I am struggling everyday to experiment for solutions.

I do think you should be a little bit more open-minded to the possibility that @keean and I might actually find a new quandrant in the PL design space that is interesting and worthwhile. Seems when you do not understand some of my technological arguments or presume they’re irrational because they don’t align with your experiences, you jump to presuming that I am just a complete waste of your time. And it shows in the veiled indignation of your reactions. I wish we could keep our interactions on a productive level. Maybe our personality types, background and current life situations don’t mix well. I dunno. But really I do not want to be in constant state of animosity with you. Never did I aim for that. The conflict seemed to be amplified when I expressed how frustrated I am with the Power Rangers, SJW culture that has flooded our industry. And after 20 years that culture still hasn’t produced the PL I think I want. We’ll see how this plays out...

Again I would be delighted if someone else grabs the design ideas here and implements them faster than I can attempt to in my current situation in life.

History of this tangential discussion:

https://github.com/keean/zenscript/issues/49#issuecomment-650416801

https://github.com/keean/zenscript/issues/30#issuecomment-644488865

https://github.com/keean/zenscript/issues/30#issuecomment-643711198

https://github.com/keean/zenscript/issues/50#issuecomment-650441788

https://github.com/keean/zenscript/issues/30#issuecomment-644135568

[NodixBlockchain]

Or does anyone have a better idea for how to reconstruct this information in a suitable format that meets the above objectives?

To me it doesn't seem too hard to rebuild this from the mails, but it miss some of the first posts, even if comparing from the html save you have it seems i have most of it.

But it has the poster name in the 'mail from', dates, id, i already did this sort of thing to restore crashed forum databases or convert forum databases.

Again I would be delighted if someone else grabs the design ideas here and implements them faster than I can attempt to in my current situation in life.

I'm really not far from giving a shot a this :)

[shelby3]

@NodixBlockchain

I'm really not far from giving a shot a this :)

Do you need to be compensated monetarily? Do it with a high-level programming language would probably be the easiest. Have you studied the Github API to see what is possible? I was going to manually start doing it otherwise, while I still have the copy of the deleted thread in a browser tab.

Feel free to email me: shelbymoore3@protonmail.com

[NodixBlockchain]

I'm really not far from giving a shot a this :)

Do you need to be compensated monetarily? Do it with a high-level programming language would probably be the easiest. Have you studied the Github API to see what is possible? I was going to manually start doing it otherwise, while I still have the copy of the deleted thread in a browser tab.

To process the mails i would probably go for something like php or perl which are good at dealing with text processing, eventually javascript or python, but i don't know python much, my perl is a bit rusty but it's not extremely difficult to parse either.

The github api doesn't seem to difficult either

https://developer.github.com/v3/issues/comments/#create-an-issue-comment

[shelby3]

@NodixBlockchain email me: shelbymoore3@protonmail.com

[NodixBlockchain]

@NodixBlockchain email me: shelbymoore3@protonmail.com

sent the mail

[shelby3]

@NodixBlockchain email me: shelbymoore3@protonmail.com

sent the mail

I appreciate the effort you put into this, but unfortunately email data is not going to work well, especially not for my posts because I make so many edits. The email data does not include the edits as you can note on this screen shot comparison of your recovery versus my live copy of the page that was deleted:

https://imgur.com/a/B27XiHk