Kotlin / KEEP

Kotlin Evolution and Enhancement Process
Apache License 2.0
3.37k stars 361 forks source link

Encapsulate successful or failed function execution #127

Closed elizarov closed 3 years ago

elizarov commented 6 years ago

This issue is for discussion of the proposal to provide a Standard Library class to encapsulate successful or failed function execution.

Proposal text: result.md

pakoito commented 6 years ago

Re. flatMap and the current API: even if the Scala code looks similar it doesn't do the same. Don't fall in that trap. It's rewritten behind the scenes the same way coroutines do to a chain of operations. The return type is a Try<List<T>>, so to be equivalent your example would have to return SuccessOrFailure<List<T>>, and the same has to be returned from the other methods.

So taking the original code:

fun getURLContent(url: String): List<String> {
    val url = parseURL(url) // here parseURL returns URL, throws on failure
    val connection = url.openConnection()
    val input = connection.getInputStream()
    val source = Source.fromInputStream(input)
    return source.getLines()
}

What happens when parseURL(url), openConnection(), getInputStream() return SuccessOrFailure<T>? Do you have to use the getOrNull and nullcheck it every line? Then you lose information about the exception, and makes it more verbose. If you have to getOrThrow and try/catch every line then you don't have any improvement over the current state of the world with normal exceptions. I'll write 4 snippets each with a different way of chaining these operations and are all valid (save for bugs :P) with the proposed API:

fun getURLContent(url: String): SuccessOrFailure<List<String>> {
    val url = try { 
                  parseURL(url).getOrThrow()
                } catch (e: ParseException) { 
                  return SuccessOrFailure.failure(e)
                }
    val connection = try {  
                  url.openConnection().getOrThrow()
                } catch (e: ConnectionException) { 
                  return SuccessOrFailure.failure(e)
                }
    val input = connection.getInputStream().getOrThrow() // anything thrown here is non-recoverable
    val source = try {  
                  Source.fromInputStream(input).getOrThrow()
                } catch (e: IOException) { 
                  return SuccessOrFailure.failure(e)
                }
    return SuccesOrFailure.success(source.getLines())
}
fun getURLContent(url: String): SuccessOrFailure<List<String>> {
    val url = parseURL(url).getOrNull()
    if (url != null) {
      val connection = url.openConnection().getOrNull()
      if (connection != null) {
        val input = connection.getInputStream().getOrThrow()
        val source = Source.fromInputStream(input)
        if (source != null) {
          return SuccessOrFailure.success(source.getLines())
        } else  {
          return SuccessOrFailure.failure(SomethingException("1"))
        }
      } else {
         return SuccessOrFailure.failure(ConnectionException())
      }
    } else {
      return SuccessOrFailure.failure(ParseException())
    }
fun getURLContent(url: String): SuccessOrFailure<List<String>> {
  return runCatching {
      val url = parseURL(url).recover { if (it is ParseException) { return SuccessOrFailure.failure(it) } else { throw it }  }.getOrThrow()
      val connection = url.openConnection().recover { if (it is ConnectionException) { return SuccessOrFailure.failure(it) } else { throw it }  }.getOrThrow()
      val input = connection.getInputStream().getOrThrow()
      val source = Source.fromInputStream(input).recover { if (it is IOException) { return SuccessOrFailure.failure(it)  } else { throw it }  }.getOrThrow()
      source.getLines()
  }
  .recover { if (it is InputStreamException) { throw it } else { return SuccessOrFailure.failure(it) }  }
}
fun getURLContent(url: String): SuccessOrFailure<List<String>> {
    try {
      val url = parseURL(url).getOrThrow()
      val connection = url.openConnection().getOrThrow()
      val input = connection.getInputStream().getOrThrow()
      val source = Source.fromInputStream(input).getOrThrow()
      return  SuccessOrFailure.success(source.getLines())
    } catch (e: ConnectionException) { 
        return SuccessOrFailure.failure(e)
    } catch (e: ParseException) { 
        return SuccessOrFailure.failure(e)
    } catch (e: InputStreamException) {
       throw e // so it doesn't get caught by IOException
    } catch (e: IOException) { 
       return SuccessOrFailure.failure(e)
    }
}

None of the snippets is an improvement over the current situation of try/catching, and the API is ripe for abuse as it doesn't provide a clear, unique way of chaining SuccessOrFailure. The non-local returns required for coroutine rewrites also enable more possibilities for breaking control flow, and will create a copy of frequently called APIs on every callsite.

Every company, library, and employee will have their own preferred style, which means every time you come to a file using SuccessOrFailure it will look different and have different control flow that's not the one JetBrains had in mind for this proposal. You're trading learning a concept once, for all libraries, with having to read an implementation to understand its behavior every time.

If you let the exception propagate then your API is lying to you, it's not a SuccessOrFailure<List<T>>, it's a hidden exception instead like most Java APIs do. It's not an improvement, it's a step back with a new broken way of writing APIs.

You have the body of work that Arrow has done around this already. The flatMap way of chaining constructs is principled and shared under other names like bind, andThen and selectMany across JavaScript, C#, Java, TypeScript, Rust, Scala, Clojure, Haskell, F#, OCaml, ReasonML, a handful of libraries for each, and even other parts of Kotlin. It's well understood that flatMap chaining is not equivalent to exception propagation even with checked exceptions, nor the two snippets provided are the same and the alternatives described above are even more verbose than regular exceptions.

We have the implementation, the test suite, the documentation, entire communities of untyped languages that understand the concept and enough examples across all of them. I don't understand the pushback.

raulraja commented 6 years ago

https://github.com/Kotlin/KEEP/blob/success-or-failure/proposals/stdlib/success-or-failure.md#appendix-why-flatmap-is-missing just want to note that this section is not accurate. Direct style is not equivalent to monadic comprehensions unless in the Id Monad. All other monads take care of their effects including those capable of failure without blowing the stack throwing exceptions. From all the things Kotlin has gotten from Scala monad comprehensions is the big one missing. The entire Scala community even those uninterested by FP like for comprehensions. I think a lot of people in the Kotlin community would benefit from safer error handling and generalized syntax for types that structurally include a way to flatMap like List, Deferred, Observable,...

pakoito commented 6 years ago

Someone has pointed me that if you look closely, parseURL(url).mapCatching { url -> url.openConnection().getOrThrow() } is equivalent to parseURL(url).flatMap { url -> url.openConnection() }.

Given that Raul and I have been the first ones to reply, it's worth mentioning that we don't have any agenda with this topic other than trying to help with the API based of experience with similar constructs, and what's happened in the past in other communities. For example, the JS community ignored a similar API design correction citing "aesthetics" in Promises/A+ that years later they're still paying for, and ended up spinning up the whole fantasyland ecosystem and a set of alternative Promise libraries and extensions trying to fix this mistake.

If we're not being helpful or insightful, we'll just add SuccessOrFailure#flatMap as an extension function in a single function library and everyone will get the API what they want :D

elizarov commented 6 years ago

@pakoito Let me try to answer your concerns

What happens when parseURL(url), openConnection(), getInputStream() return SuccessOrFailure<T>?

We explicitly discourage writing functions that return SuccessOrFailure<T>. Don't fall in the trap of thinking that we are proposing to add Arrow's Try class in disguise into the Kotlin Standard library.

That is why we don't have to answer this "what happens" question nor do we have to write any of the verbose alternatives you are suggesting. Moreover, we explicitly discourage using SuccessOrFailure to catch some exceptions, while letting others fall through. SuccessOrFailure is explicitly designed to encapsulate any failure in the function execution.

We do encourage using domain-specific classes in case where you need more fine-grained control over failures that need to be caught and the ones that need to go through. We are not planning to eat Arrow's bread here and that is one the reasons we gave such a long SuccessOrFailure name to the stdlib primitive. We purposefully avoided taking over names that functional libraries are using. You are welcome to continue Arrow's primitives without confusing anyone who is reading the code on the intended semantics of those primitives.

I will expand the corresponding section in the KEEP to better explain the similarities and differences between monadic approach and the use-cases that SuccessOrFailure is designed to address.

elizarov commented 6 years ago

@raulraja The KEEP explicitly lists it pretty narrow goals. Introducing alternative to exceptions for error-handling is out-of-scope of this KEEP. This KEEP is narrowly focused on the problem of dealing with exceptions in concurrent and functional code.

elizarov commented 6 years ago

Let me give the summary of various way to represent errors in Kotlin code:

elizarov commented 6 years ago

@pakoito

Someone has pointed me that if you look closely, parseURL(url).mapCatching { url -> url.openConnection().getOrThrow() } is equivalent to parseURL(url).flatMap { url -> url.openConnection() }.

That is correct. One can easily write flatMap as an extension, but since we discourage using SuccessOrFailure as return type of functions, there is no point in having it in the standard library as its use would contradict the design goals behind SuccessOrFailure.

elizarov commented 6 years ago

@pakoito

We have the implementation, the test suite, the documentation, entire communities of untyped languages that understand the concept and enough examples across all of them. I don't understand the pushback.

That is great. There is no pushback. I, personally, really like the way Arrow community is working and the resulting functional libraries and I would like to see it thrive and expand. It works really well as a separate library that people can use if they develop code in mostly functional style and I see no point in stealing any of the Arrow's functionality into the Kotlin Standard Library. Arrow is already quite popular and works really well the way it is.

From my side, I will try to give a better explanation on differences between Arrow's Try and the SuccessOrFailure that this KEEP is proposing and to focus more attention on the additional benefits that Try is bringing onto the table to encourage people who are interested in those to use Arrow library. I did not intent this appendix to be perceived as a pushback on Arrow. I'll need to do a better writing job there.

ktkit commented 6 years ago

Thank you, Roman. It's great this type has reached the stage of KEEP. Yet still, I strongly believe its value is underestimated.

Let's consider a simple scenario. We've designed the following function that throws an exception, having in mind the user code just propagates it on the higher level. So we document its dangerous behavior and voila:

/** Returns [URL].
@throws SomeException... */
fun parseURL(url: String): URL {
  // parses a string and may throw an exception in case of a malformed url
  return theUrl
}

Then a user calls it in some place where error propagation is expected:

fun getURLContent(url: String): List<String> {
  val url = parseURL(url) // it throws but we expect it
  // ... process content
  return content
}

But some time later the user sees this usage of parseURL function which obviously returns URL, and copy-pastes it to that function:

fun showURLContent(url: String) {
  val url = parseURL(url)  // oops .. Application is terminated due to an unexpected error
  // ... process content
  textField.text = content 
}

And oops. It's slipped from UnitTests and real users experience the bug :(

Such scenarios happen all around because the relation between method signature, its returned type and the fact that it's dangerous, is held only on short-term memory of the developer who uses it. No other hints available.

Kotlin removed an explicit Java-like method annotation throws (thank god!) but provides nothing safe in exchange (Null-safety is cool! But it's about references, not a behavior of functions). And so we have the same "billion-dollar mistake" problem that we had in Java with null references. But now in Kotlin, and on the higher level of functions. Figuratively speaking it's the same as holding dynamite in candy wrap. Looks sweet, touch it, and you'll find yourself in chocolate up to the ears :)

And lo! The SuccessOrFailure type which literally introduces the new paradigm of exception handling:
Any function that may throw an expected exception returns SuccessOrFailure object.

fun parseURL(url: String): SuccessOrFailure<URL> {
  // parses string and may throw an exception in case of a malformed url
  return successOrFailureUrl
}

And since then nothing wrong can happen.

fun showURLContent(url: String) {
  val url: URL = parseURL(url).onFailure{return}.getOrThrow() // continue safely with a URL object
  // ... process content
  textField.text = content
}

And if one needs an error propagation behavior, the following explicit alternative to the parseURL can be defined:

fun parseURLThrowing(url: String): URL {
  // parses string and may throw an exception in case of a malformed url
  return theUrl
}

Or the safe version can be used this way:

fun getURLContentThrowing(url: String): List<String> {
  val url = parseURL(url).getOrThrow()
  // ... process content
  return content
}

And since the implementation of the SuccessOrFailure class is highly optimized, it costs nothing to box-unbox this object the way above.

I expect objections from developers who got used to the "unsafe" style and/or has a bunch of code written that way. So yes, it's a tough decision to make. But I believe it's worth it. Anyway, it would be great to have this type in Standard Lib. But you sell it for 1% of its real value.

P.S. In case of reconsidering the role of the SuccessOrFailure class, the argument that this class "can abuse as a return type of a function" becomes not effective. And the name "Result" and other shorter names can still be under discussion.

P.P.S: The unsafe method signatures is the reason of the discussions about deprecation of Java Optional.get()

Java 8 was a huge improvement to the platform, but one of the few mistakes we made was the naming of Optional.get(), because the name just invites people to call it without calling isPresent(), undermining the whole point of using Optional in the first place. (If this was the worst mistake we made in such a big release, then we did pretty well.)

(I'm Markus M from this discussion)

The bottom line is: This type is so elegant exception handling solution that it by design encourages developers to use it as a return type from functions that throw expected exceptions.

pakoito commented 6 years ago

We explicitly discourage writing functions that return SuccessOrFailure

So if I understand this correctly this KEEP exists almost exclusively to address error handling in coroutines, as its current design requires try/catching and it doesn't compose, which is why we ended up extending it on Arrow. Why does SuccessOrFailure have to be added into the language instead of kotlinx.coroutines directly then?

My concern with adding it to the language is that even if you put documentation, give a stern talk at kotlinconf, put a linter in IJ, a compiler warning and post about it on several blogs, I suspect many people will ignore it and still use them as function returns because that's the perceived reason a Result-like type has to exist. Your examples already include Deferred<SuccessOrFailure<T>> and List<SuccessOrFailure<Data>> as function returns that escape the scope of coroutines.

The other concern I have is that the API is too barebones just for aesthetical purposes. We know from user data gathered over the past year that most users come ask about a function sequence() that goes from List<SuccessOrFailure<Data>> to SuccessOrFailure<List<Data>>, a shape that's easier to work with in their heads. Same as flatMap they'll reimplement it in non-canonical, untested ways, or will get it from a library. People who are not willing to learn map, flatMap, fold and sequence already use when and for loops to recreate the same functionality; which doesn't mean that you can still provide the canonical functions for everyone else. There are no downsides, just aesthetical reasons, and the community will not necessarily use SuccessOrFailure the same way this KEEP intends because the lack of helpers forces them to look for their own solutions.

As @ktkit said if this is a construct added for the language, might as well go all-in the first time around and copy Rust's Result or any other proven body of work you prefer. At Arrow we're eager to deprecate Try if it's added to the language, proper. If that is not the intention of the KEEP, a couple of extension functions may suffice instead.

elizarov commented 6 years ago

@pakoito The primary use-case for this SuccessOrFailure class, as explicitly spelled out in the corresponding KEEP is indeed error handling in coroutines. We have to put it into the Kotlin Standard library because Continuation interface from the Kotlin Standard Library depends on it. We cannot put it into kotlinx.coroutines, but we can put it into kotlin.coroutines. However, since we've identified a number of use-cases beyond coroutines (and they are all listed in the KEEP) we decided to put into kotlin package for now.

We very much share your concern that people will use it as a return type and we'll try our best to discourage people from doing so.... we cannot entirely prevent bad code, though.

Giving additional extensions for things like List<SuccessOrFailure<T>> was raised as an open issue. I'll add it as an open issue to the KEEP, but we'll need to make sure that we are not encouraging abuse of SuccessOrFailure when doing so. In this respect, barebones API is done on purpose. Again, we don't want to eat Arrow's bread. This API is designed for very narrow use cases, not as something you'll use all over the place in your application.

We could add something broader into the language in the future, but that is definitely out of the scope of this KEEP. Don't deprecate your Try please. We are not in a position to replace it yet. Replacing all the exception-handling patterns you can do with Try and putting them into the language in some shape is extremely complicated endeavor. We have not even narrowed down any potential direction we might take on this road (and there are many), yet alone figured out appropriate design.

raulraja commented 6 years ago

The question for me is why isn't this for a more generic Either in the std lib and let users decide how they want to use it. I'm in favor of removing types from Arrow that surface in the stdlib in some kind of compatible shape with the functional combinators. If we had Either in the std lib you could have in Deferred the operations you need to handle errors even when ops go async without blowing the stack:

fun <A> Deferred<A>.attempt(): Deferred<Either<Throwable, A>>
fun <A> List<Deferred<A>>.sequence(): Deferred<List<A>> // shortcircuits on first error 

val op1: Deferred<Int> = async { 1 }
val op2: Deferred<Int> = async { throw RuntimeException("can't fetch 2") }
val op3: Deferred<Int> = async { 3 }

val result: Either<Throwable, List<Int>> = 
  listOf(op1, op2, op3).sequence().attempt().await()

result.fold(::showError, ::doWithSuccess)

I think the issue with error handling in async stuff is that we instead encourage people to do things that break composition when exceptions are thrown, at least a lot of the code I see with coroutines looks like calling await anywhere is fine forcing then again to try/catch or it blows with an exception.

When you say: | We explicitly discourage writing functions that return SuccessOrFailure I think that leads to people having to handle errors, async/sync etc explicitly on each layer or resort to throwing exceptions and back to the original style of try/catch.

val result: List<Int> = try {
   listOf(op1,await(), op2.await(), op3.await())
} catch (e: Throwable) {
  emptyList<Int>()
}

I think an Either type for the standard library that allows others like Coroutines solve their own problems would be more appropriate specially when there is already a proliferation of Result types in many libraries and a bunch of custom ones that people ad-hoc put in their code bases and none of them solve the async biz.

fvasco commented 6 years ago

A personal consideration about @ktkit Optional mention.

Optional does not fix the billion-dollar mistake, it is only a verbose workaround. Every Java method can return null instead of Optional, so the mistake remains unchanged. To fix this issue is required a language change.

Likewise for SuccessOrFailure, there is no way to guarantee a result instead of an exception.

In the previous example:

fun parseURL(url: String): SuccessOrFailure<URL> {
  // parses string and may throw an exception in case of a malformed url
  TODO() // return successOrFailureUrl
}

So SuccessOrFailure as return type is trustworthy like Optional in Java.

elizarov commented 6 years ago

@ktkit Thanks for taking time to explain the dangers of exceptions. Let's dissect your example a little further:

/** Returns [URL].
@throws SomeException... */
fun parseURL(url: String): URL

We really encourage Kotlin programmers NOT to declare such functions because of all the problems with exceptions you've outlined. If you have signalling exception that is supposed to be caught in your code and handled locally you should not have used exception for that in the first place. An idiomatically designed Kotlin API for this case should be defined like this:

/** Returns [URL] or `null` if it cannot be parsed. */
fun parseUrlOrNull(url: String): URL?

With this signature you get all the nice typesafety and extensive support from Kotlin compiler and standard library, which allows you to write succinct, yet correct and safe code like this:

fun showURLContent(url: String) {
  val url: URL = parseUrlOrNull(url) ?: return // continue safely with a URL object
  // ... process content
  textField.text = content
}

It is out of the scope of this KEEP to design similar mechanics for wider classes of failure that a class like SuccessOrFailure might represent, that is why we discourage writing functions that return SuccessOrFailure object. Please, use nullable type in the code like that.

elizarov commented 6 years ago

@raulraja We are painfully aware of exception handling problems with concurrent code and how it is very hard to write correct code using async (without running into very subtle problems). We are trying to address it in kotlinx.coroutines via a more structured concurrency approach, see https://github.com/Kotlin/kotlinx.coroutines/issues/410 for details.

elizarov commented 6 years ago

UPDATES: @pakoito @raulraja @ktkit

  1. I've expanded Appendix with a more fleshed out comparison of SuccessOrFailure and Try to explicitly address the differences in how they work and how they are used.

  2. I've added Additional APIs for collections into the list of open questions.

  3. I've worked out future direction for Integration into the language to show a potential path on how one day we might realize a Kotlin language mechanism of finer-grained control of exceptions that could replace Java checked exceptions with a safer and more explicit approach in Kotlin style.

raulraja commented 6 years ago

I reread the Appendix and these are not equivalent:

def getURLContent(url: String): Try[Iterator[String]] =
  for {
    url <- parseURL(url) // here parseURL returns Try[URL], encapsulates failure
    connection <- Try(url.openConnection())
    input <- Try(connection.getInputStream)
    source = Source.fromInputStream(input)
  } yield source.getLines()
fun getURLContent(url: String): List<String> {
    val url = parseURL(url) // here parseURL returns URL, throws on failure
    val connection = url.openConnection()
    val input = connection.getInputStream()
    val source = Source.fromInputStream(input)
    return source.getLines()
}

The first returns Failure(e) whereas the second just throws and blows the stack. The whole point of the result types is that they are an Algebraic Datatype with multiple potential states. In this case Success(lines) and Failure(e) they are not equivalent and flatMap which is what for comprehensions desugar to is always biased toward success. The user is not forced to handle the exception but her function does not throw an exception either. Case 1 is exception free.

I also find the encouragement of losing information when handling errors bad practice suggesting that fun parseUrl(url: String): URL? is better. That type signature causes loss of information and models error handling where the type of error should be known or at least a Throwable with a type that it's used to model the absence of a value. Instead we are encouraging to disregard exceptions and errors and model them as absence. This style of narrowing to ? forces users to always be explicit about error handling in their function bodies to go from E -> A? and that is not necessary when the data type already provides an error context like for example Deferred, Observable, Either, Try, etc. These are all types which have an error context and don't need to have users explicitly try catch around them ever. There is a way to compute over these types in a sane way which is via comprehensions that delegate to flatMap. I understand that is not the issue this KEEP addresses but really comprehensions allow for easy composition and imperative style and it works in the same way in sync and async code.

Here is the example above with a pseudo keyword bind that I'm making up to tell the compiler to suspend and use a coroutine context to invoke flatMap behind the scenes until it completes binding it's result to the left hand side. (alternatively the compiler could just desugar the tree to a flatMap chain)

fun doUrlStuffSync: Try<Array<String>> {
  bind url = parseURL(url) //returns Try<Url>
  bind connection = Try { url.openConnection() }
  bind input = Try { connection.getInputStream }
  val source = Source.fromInputStream(input)
  return source.getLines()
}

Now here is the difference with deferred:

fun doUrlStuffAsync: Deferred<Array<String>> {
  bind url = parseURL(url) //returns Deferred<Url>
  bind connection = async { url.openConnection() }
  bind input = async { connection.getInputStream }
  val source = Source.fromInputStream(input)
  return source.getLines()
}

Same syntax if we have A?#flatMap for values returning ? and same syntax for List since there is List#flatMap as well.

Same syntax for all types. Folks don't need to be explicit about exception handling everywhere and types are still forcing them to deal with the exceptional case at some point without resorting to try/catch also we are not encouraging flatMap chains but imperative style top to bottom.

The problems of Either, Try, Deferred etc in terms of composition and sequential computation are the same. All these types compose with flatMap, short-circuit in their first error and carry internally an error context that flatMap ignores. If we end up adding special syntax for all of them as suggested in:

|User throws NotFoundException, MalformedNameException

we are gonna end up with special cases for each type and it's going to be a mess composing values like Deferred<Either<MalformedNameException, User>> where you have to resort to a bunch of different machinery to reduce those boxed values to the values they contain within.

pakoito commented 6 years ago

Re. the integration into the language section, you already have a primitive with its own syntax that handles errors with propagation: coroutines and the Deferred return type. If you manage to make them cheaper, or even better, depend on the implementation of an operator similar to flatMap like @raulraja described above you won't need this new syntax.

If we continue through this path unpacking a Deferred<SuccessOrFailure<NewErrorType<T?>>> is going to require lots of special syntax baked into the language that'll be hard on newcomers. This is a common issue in monad transformer stacks, where you can end up with dumpsterfires like ReaderT<ForIO, Environment, EitherT<ForIO, Option<T?>>> that without implicit arguments or coherent typeclasses are quite cumbersome.

With the flatMap operator that burden is shifted to library and language implementors instead, as it unifies the behavior. The code will still use the "fast path" version of the coroutines machinery behind the scenes, with the same syntax.

Wasabi375 commented 6 years ago

I don't see why you would get a Deferred<SuccessOrFailure<NewErrorType<T?>>> anywhere. The example with deferred does not force this on you.

val outcomes1: List<T> = deferreds.map { it.await() }
// ...
val outcomes3: List<SuccessOrFailure<T>> = deferreds.map { it.awaitCatching() } 

You can still use await instead of awaitCatching. As SuccessOrFailure should not be used as a return type I don't see why you would ever get a SuccessOrFailure<NewErrorType>. The usage of SuccessOrFailure is totally optional, you as the user only use runCatching at the place where you actually want to handle the error so you don't need to worry about stacking error types as this one only gets used temporarily to handle the error. And as state in the proposal, if you need complex error handling you should look to different options anyways

If your code needs fine-grained exception handling policy, we'd recommend to design your code in such a way, that exceptions are not used at all for any kinds of locally-handled failures (see section on style for example code with nullable types and sealed data classes). In the context of this appendix, parseURL could return a nullable result (of type URL?) to indicate parsing failure or return its own purpose-designed sealed class that would provide all the additional details about failure (like the exact failure position in input string) if that is needed for some business function (like setting cursor to the place of failure in the user interface). In cases when you need to distinguish between different kinds of failures and these approaches do not work for you, you are welcome to write your own utility libraries or use libraries like Arrow that provide the corresponding utilities.

Wasabi375 commented 6 years ago

@elizarov btw, the awaitCatching functions are not in the API details

elizarov commented 6 years ago

@raulraja I'll try to answer some of your concerns:

I reread the Appendix and these are not equivalent: ... The first returns Failure(e) whereas the second just throws and blows the stack.

It looks like the following sentence from Appendix answers it:

If callers of this function need an encapsulated failure, they can always use runCatching { getURLContent(url) } expression.

With respect to this:

I also find the encouragement of losing information when handling errors bad practice suggesting that fun parseUrl(url: String): URL? is better.

I'll quote this part:

If your code needs fine-grained exception handling policy, we'd recommend to design your code in such a way, that exceptions are not used at all.

The key part here is "if your code needs...". Not all cases are the same. Sometimes we need to "blow up the stack", because we don't plan any local, business-specific handling, and that is where exceptions come in handy, since they contain all the vital debugging information (stack trace and message). But none of that debugging information is useful for business logic. If you are planning to handle URL parsing failure in the business part of your code (what I call "locally"), then having exception at hand does not give you any additional value. Quite contrary, you loose tons of performance constructing this exception only to get a single bit of useful information out of it -- whether parseURL had failed or not. Nullable types are much better here.

Same syntax for all types ...

You are suggesting to unify all the error-containg types, but we are trying to do exactly the opposite. We are trying to discourage usage of exception-containing types in the code. We want to have a world where Kotlin code has as few uses of SuccessOrFailure and Deferred types as possible. These types are noise. Programmers should be focused on the business logic of their code, while error-handling and concurrency aspects of their code should be optional layers -- they should not obscure or encumber the main logic of the program, they should not force the programmers to write the main logic of their code in any different way, but we also need to make sure that the shortest code you write is still safe, correct, and does not loose any resources or exceptions (we are still far from this bold goal, but we are trying to get there).

You can see it in the design of all the Kotlin things. That's why, for example, we have suspend fun(...): T concept instead of fun (...): Deferred<T> as many other languages chose to do, because we believe that programmer should not be forced to change the way they write their business logic because they now have some asynchronous operations in their code.

UPDATE: ^^^ this also answers @pakoito concern. In short, we don't want to encourage things like fun foo(): Deferred<SuccessOrFailure<NewErrorType<T?>>> in Kotlin code (though we cannot prevent people from writing them, of course). It should be suspend fun foo(): T? and nothing more.

elizarov commented 6 years ago

@Wasabi375 awaitCatching is an example. It cannot be part of the Kotlin Standard Library, because awaitCatching is an example of extension you could define on Deferred class which is defined in kotlinx.coroutines library.

pakoito commented 6 years ago

Okay, I understand the intents and purposes of this KEEP and my concerns have been heard. I'll help spread the word not to use SuccessOrFailure as a function return, add some helpful extfuns to arrow-core and promote Try/Either as the choice for public APIs instead :D If there's something else I can do to help just tell me!

ktkit commented 6 years ago

@elizarov Thanks Roman, I like your idea of dissection. Let's just take the whole pie, i.e. the handling of functions that are supposed to fail under specified conditions in which case they should signal it somehow to the caller (in short, fallible functions), and review this subject in evolutionary perspective.

Pre-SuccessOrFailure era:
Available scenarios:

  1. A failure is univocal or extra info isn't relevant. Example: Division by zero. Solutions: a. If the returning type is non-nullable: return nullable type T?, where null indicates a failure. b. Otherwise return Optional, where empty indicates a failure.
  2. Info about a failure is/may be relevant. Example: parseURL which failure may indicate what had gone wrong. Solutions: a. Throw an exception that carries relevant info, hoping it's caught somewhere in the stack. b. Devise a (or use a 3-rd party) custom class or hierarchy of classes to carry this info.

Notes: 1a is the case you've mentioned in your response. 2a is the case, you agreed, is "dangerous". However, most of the developers did not append the name of such functions with "OrThrow" (or alike) since there wasn't "non-dangerous" alternative. 2b is seen as an ancestor of this KEEP subject.

Post-SuccessOrFailure era:
Available scenarios:

  1. A failure is univocal or extra info isn't relevant. Solutions: a. If the returning type is non-nullable: return nullable type T?, where null indicates a failure. b. Otherwise return SuccessOrFailure instead of Optional.
  2. Info about a failure is/may be relevant. Solutions: a. Return SuccessOrFailure since it's the efficient functional value-exception container.

Notes: 1a is the only case left where handling of a failure is still more efficient than the SuccessOrFailure way, since specific info about a failure isn't required. The throwing version of a fallible function that relies on catching can still exist but with some intelligible name extension like "OrThrow".

For all other cases, the SuccessOrFailure type represents an evolutionary leap to safety and simplicity.

Please, correct me if I'm wrong.

elizarov commented 6 years ago

@ktkit There is nothing efficient about SuccessOrFailure in case of a failure. On a failure it contains an exception that is extremely expensive to create. It should not be used for case where you expect to have a failure condition that would be handled by some business logic. Exceptions should be only used for truly exceptional cases of the kind that crash your application and are usually handled somewhere on the top level, that is why they are designed to propagate automatically and to carry all the metadata for debugging.

For cases where you expect some kind of potential error in function execution you should use either nullable types or domain-specific sealed class hierarchies. If you write all your code in a monadic style, then you should use the corresponding monadic libraries (like Arrow) which integrate the corresponding monads.

raulraja commented 6 years ago

@elizarov Would mapCatching and other similar methods catch all Throwables or would it perform some kind of discrimination like Scala does with NonFatal https://www.scala-lang.org/api/current/scala/util/control/NonFatal$.html ?

ktkit commented 6 years ago

@elizarov I admit that the fact that the default Throwable.fillInStackTrace method is expensive slipped out of my attention. Now i see, you've got a fair reason to discourage from using this type in cases when failures in a fallible function happen quite often and thus induce pressure on CPU. And this fact, I believe, should be explained in the API docs as the reason why you discourage from abusing this type since it isn't that obvious.

However, a couple of optimizations can eliminate this adverse cost effect.

Let's review scenarios of handling a fallible function in a new perspective:

  1. A failure is univocal or extra info isn't relevant: a. If the returning type is non-nullable b. If the returning type is nullable
  2. Info about a failure is/may be relevant: a. The exception stack trace is relevant b. Local info about the failure is enough

Solutions:

So maybe instead of or in complement to discouraging developers from abusing this type, informing them about alternatives is not a bad idea?

elizarov commented 6 years ago

@raulraja We don't have anything similar to NonFatal in Kotlin at the moment and designing something like that is complicated endeavor that is outside of the scope of this KEEP, so all xxxCatching functions just catch all Throwable exceptions. That is just another reason why SuccessOrFailure should not be used outside of the narrow use-cases it is designed for. My own experience in enterprise shows that distinction between "fatal" and "non-fatal" is extremely domain-specific and business-specific. Hardcoding it into the language (standard library) would definitely work against the "general purpose" nature of Kotlin. Having it defined in 3rd party libraries is perfectly fine, though, since those are domain-specific by definition.

I've prototyped a version of SuccessOrFailure that would let you a specify a (subtype) of exceptions that are caught, but it does not look very well in Kotlin's generics at this moment (there is no ability to specify default generic type, nor a way for partial type specification). We have to defer it to potential future enhancements. I would rather fold SuccessOrFailure into the language in the future (see updated section on this possibility) as a better replacement of Java's checked exceptions that works well with higher order function (unlike Java's checked exceptions), since it is actually expressed and implemented as a return type of a function. That would be a major enhancement in the language, though.

pakoito commented 6 years ago

nor a way for partial type specification

Check this, see if it helps: https://github.com/arrow-kt/arrow/blob/master/modules/core/arrow-instances-core/src/main/kotlin/arrow/instances/either.kt#L114-L125

elizarov commented 6 years ago

@pakoito I've tried various tricks, but it, for the purpose of this KEEP, it brings more pain than benefits. We have only one use-case where you could benefit from catching a specific subset of exceptions runCatching<IOException> { ... }.onFailure { ... }, but I could not make this pattern to work in this clear and easy-to-read work without making other use-cases harder to read. Further increasing API weight just for this one use-case does not seem worth it.

pakoito commented 6 years ago

Maybe splitting the API to runCatchingPartial for this case?

You also have your first proposal of misuse: https://github.com/kittinunf/Result/pull/38#issuecomment-403727685

elizarov commented 6 years ago

@pakoito You can define runCatchingPartial in your library if it is needed. I'm hesitant to add something like that to stdlib for SuccessOrFailure. It does not work that well when the type is not parameterized by the type of exception. You'd be better off using something like Result<T, E> from @kittinunf if you have the need in catching only subset of exception (you'll have the type of encapsulated exception represented in the type).

elizarov commented 6 years ago

@ktkit We're looking at error handling from two different perspectives and I don't see a clear way to reconcile this two views. When I'm systematically dissecting error handling I look at it like this:

  1. The failure will be handled by the business logic in the caller code.
  2. The failure will be handled by the global handler with the purpose of providing info for developers.

Solutions:

  1. Return nullable type or a business-specific sealed class. No need for exceptions here as all the exception metadata (like stacktrace) is of no use to business logic that decides what to do.
  2. Throw an exception.

You see, there is no place to return SuccessOrFailure as a result type among those solutions.

ktkit commented 6 years ago

@elizarov Great. Now, seems, we have all missing parts. The pie (fallible function), the pie baker (function designer) and the pie eater (function caller). You happened to present the subject from the perspective of bakers (designers). As they reason from the point of the actual value of a failure info and thus where, in the call stack, it should be handled. I happened to present the subject from the perspective of eaters (callers). As they decide ad-hoc whether the failure info is irrelevant or relevant and in what amount.

Your point is totally valid (when you are an API designer) but slightly idealistic when it comes to its application. Just because you can't predict the exact intention and scenario in which the fallible function is called. Let's take fun findUserByName(name: String): UserResult from your example. Its return type is really suitable when the caller of that function is going to handle all that info. Otherwise, if s/he intends to use a success value and just to log a failure, the return type representation gets annoying.

Regarding the case:

  1. The failure will be handled by the global handler with the purpose of providing info for developers. -> Throw an exception.

It's safe to do only if the designer of such function guarantees the thrown exception is caught in the designated place. Which s/he can guarantee only having total control over the code. Otherwise, even if the function name is marked with something like "OrThrow", one cannot guarantee that the caller names her/his function with such "danger marker". And that's why our code blows up from time to time. On the other hand, such cases could be perfectly covered by SuccessOrFailure, since the amount of work (expensive creation of the exception) is the same. The SuccessOrFailure as a return type is totally safe. And the caller can easyly do with it whatever wanted: handle, log, rethrow, etc at no extra cost.

Regarding the case with sealed class: Well, honestly, I don't know. You pushed me into a corner with the "expensive exception" argument. I'm pondering over a wayout.

Anyway, my intent is to help come up with the multipurpose type that enhances safety and simplicity of the language. Hope our interests coincide.

P.S: I can't help putting the metaphor. Imagine a baker who bakes and sells her products in a small cozy cafe. Sometimes visitors want to eat the cake on the spot. Sometimes they eat a part of it and want the other one wrapped to eat it at home. Sometimes they just want to have a cake wrapped to eat it at home. So she can't enforce visitors to eat all in place, and can't predict which is the case. The only rational solution is to wrap each of the cakes. One might think it's expensive, but it's good for business in a long run.

elizarov commented 6 years ago

@ktkit That is the way error handling is designed in Kotlin (the language) and in its standard library. This is also the style and approach we are using in the code of Kotlin project itself. Sometimes function designers can foresee that there are different use-cases for their functions. It this case they can provide alternatives for the function. See, for example, String.toInt() (throws exception) and String.toIntOrNull() (returns nullable type) in the standard library.

Anyway, if your intent on using the function is different from the one that the original designer of this function had in mind, you can always adapt. If function throws on failure but your case requires local handing, then you can catch exceptions with try { ... } catch or with runCatching { ... } (that failure is going to be expensive, but you usually don't have a choice unless you rewrite the implementation of that function). If function returns nullable value on failure but there is nothing you can do about it locally (if, for example, this failure indicates logical error in your code or IO error that you are handling globally) then you can convert nullable types into exceptions with !! or with ?: throw MyDomainSpecificException().

We also have to live in the JVM ecosystem where it is normal to throw an exception when something goes wrong, so yes, we have to assume that all code is written this way. We assume that at the top level you always have some global exception handler that properly logs exceptions and/or reports them to developers in some other way. For example, that is how fun main operates in Kotlin. If it throws, you get the message and the stacktrace on the console for troubleshooting. In contrast, Haskell, for example, defines main :: IO (), which is a completely different beast and this enforces the corresponding monadic programming style from the language level itself.

All that I said above does not mean you cannot write your code in a different style that better suits your domain. Not only you can, you are welcome to and you should. Kotlin is designed to be flexible and universal language that can be adapted to different programming styles with the corresponding domain-specific libraries. arrow-kt.io is a very good example here.

ktkit commented 6 years ago

@elizarov Thanks, Roman. I see your point. I hope eventually there will be an efficient universal language construct, the fallible type (similar to the nullable type), that allows to carry a value along with a failure, and handle it with concise syntax like ?. As a guess, it could be based on stackless exceptions (Exception subclass with overridden fillInStackTrace), or it could be something fundamentally new. Anyway, IMHO, the type, that is the subject of this KEEP is very useful. The only thing that disconcerts is the cumbersome name. Thanks for your time.

pakoito commented 6 years ago

In contrast, Haskell, for example, defines main :: IO (), which is a completely different beast and this enforces the corresponding monadic programming style from the language level itself.

suspend fun main(varargs args: String): Unit when :D Rust already allows both Unit and Return in its main, and generates code accordingly.

elizarov commented 6 years ago

@pakoito suspend fun main is planned https://youtrack.jetbrains.com/issue/KT-17679

Redrield commented 6 years ago

When I originally voted for KT-18608, the proposal on YouTrack for a Result type before this KEEP was written, I had hoped that it could be something much more similar to Rust's Result type. What I feel is proposed by this KEEP needlessly limits the scope of what could otherwise be a very powerful tool for developers.

Take this snippet of Rust code

type Result<T> = std::result::Result<T, failure::Error> // Make all of the results parameterized by E=failure::Error

fn could_fail() -> Result<String> {
    foo()?.bar()?.baz() // In this context, ?. is binding Results. (Scala monad comprehension)
}

fn foo() -> Result<Bar>

struct Bar;
impl Bar {
    fn baz() -> Result<Baz>
}

As can be seen in could_fail, an operator is used in the place of try {...}catch(_) { return SuccessOrFailure.failure(..) } in function calls. That pattern could benefit from being a part of Kotlin for error handling the same way that ?. for null safety benefits Kotlin massively. That's something that functional companions like Arrow can't do due to their nature as third party libraries.

This proposal feels like a sad waste of what could otherwise be a very useful part of the standard library.

elizarov commented 6 years ago

@Redrield I personally love design of Result type in Rust and its integration into the Rust language. However, adding Rust-likeResult design into Kotlin would require vast changes to the Kotlin language itself with highly non-trivial consequences to Kotlin-Java interoperability (and interoperability is a very important goal at this stage of Kotlin language evolution).

It is definitely out of the scope of this KEEP (which is focused only on the standard library), but it is still on the table for future releases. One possible direction is covered in this section.

However, in order to move this work on potential future languages changes forward we first and foremost identify use-cases (with code examples) that demonstrate weak points of Kotlin's error-handling design that Rust-like Result could address. For example, this particular KEEP has quite a narrow list of use-cases that are explicitly spelled out in the text and are supported by the proposed standard library class. To do something beyond that, we'd need use-cases beyond that.

We also need a clear story of how Kotlin developers are supposed to use the feature that languages is providing them with. As of today, your Rust example in Kotlin should be, most likely, rewritten in the following idiomatic Kotlin way:

fun couldFail(): String = // means Result<String,Throwable> in Kotlin
    foo().baz() // Here Kotlin does comprehension over Try monad

fun foo(): Bar // means Result<Bar,Throwable> in Kotlin

class Bar {
    fun baz(): String // means Result<String,Throwable> in Kotlin
}

Because the original Rust code used Result<T, failure::Error> I assume that callers of this function are not supposed to actually care about what kind of failure::Error the function had failed with, but, most likely, just propagate it to the caller with ? operator to log the error for developer's trouble-shooting at the top level, so that is why, based on this assumption, I'm rewriting it with equivalent Kotlin mechanism of exceptions.

ktkit commented 6 years ago

@elizarov I still can't see why you suggest this fallible function style:

fun couldFail(): String // can throw an exception
// Or
fun String.toInt(): Int // can throw an exception

For all functions of that style that already exist (what you call JVM ecosystem) just let them be. But with the SuccessOrFailure at disposal, it's much safer to write:

fun couldFail(): SuccessOrFailure<String> // can throw an exception
// Or
fun String.toInt(): SuccessOrFailure<Int> // can throw an exception

And handle them whichever way you prefer:

fun rethrow(): String = couldFail().getOrThrow() // re-throws an exception
// Or
fun handle(v: String): Int = v.toInt().getOrElse(-1) // handle an exception locally

As for performance, there is no extra costs in both success and failed outcomes. In a success case the result won't be even boxed in a SuccessOrFailure since it's inlined (i might be wrong here, but not in the gist), and in case of a failure, the hard work has already been done anyway in the callee by creating the exception.

So actually this warning from the KEEP: This SuccessOrFailure class SHOULD NOT be used directly as a return type of Kotlin functions with a few exceptions ... could be narrowed down to: This SuccessOrFailure class SHOULD NOT be used directly as a return type only in case the failure is handled by the caller locally. Where nullable type and sealed class hierarchy could be used instead.

And if the user is aware of stackless exceptions, those can be used with SuccessOrFailure without extra costs, thus eliminating even the sealed-class hierarchy case.

I'm sorry, I know I can be a pain in the chair :)

__ By the way I've just noticed the SuccessOrFailure core class API lacks a useful, to my mind, function:

fun getOrElse(altValue: T): T

It lets write:

getInt().getOrElse(-1)
// instead of 
getInt().getOrElse { -1 }
pakoito commented 6 years ago

Those two are ambiguous because the second getOrElse could be returning a function. We desambiguated them as getOrElse and getOrDefault IIRC.

ktkit commented 6 years ago

@pakoito As I can see from REPLing the following sample, those functions are distinct, i.e. treated as overloaded. And as I can see from the current KEEP API, there's no getOrDefault.

class A<T>(val value: T) {
  fun getOrElse(v: T): T = v
  fun getOrElse(code: () -> T): T = code()
}

fun use() {
  val a = A(1)
  val x: Int = a.getOrElse(2)
  println("x=$x") // 2
  val y: Int = a.getOrElse { 3 }
  println("y=$y") // 3
}
pakoito commented 6 years ago

In val y = a.getOrElse({ 3 }), y is 3 rather than a function () -> Int as intended. Hinting doesn't fix it.

ktkit commented 6 years ago

@pakoito I do not quite understand what you're trying to get. val y: Int = a.getOrElse({ 3 }) invokes fun getOrElse(code: () -> T): T, where y is of type Int, and the argument is the function { 3 }. What am I missing?

pakoito commented 6 years ago

What I want is val y: () -> Int = a.getOrElse({ 3 }), because a holds a function of type () -> Int.

ktkit commented 6 years ago

@pakoito Charming! Thanks for the clue. When type T is a function, those overloads really become ambiguous. Although, I can't grasp the reason why. Looks like a language limitation or bug.

Anyway, the current SuccessOrFailure API lacks getOrDefault.

elizarov commented 6 years ago

@ktkit getOrDefault is indeed missing. We'll consider adding it: https://youtrack.jetbrains.com/issue/KT-25659

W.r.t. the style of fallible functions. If you are planning to handle error locally, then you should not use exceptions (nor SuccessOrFailure, for that matter). Kotlin provides two idiomatic mechanisms for locally handled errors -- nullable types and sealed classes. I am not suggesting you should write this if you plan to handle error locally:

fun String.toInt(): Int // can throw an exception, use it for globally-handled errors

I'm suggesting that you write this:

fun String.toIntOrNull(): Int? // can return null, use it for locally handled errors

Bottom-line: Don't use neither exceptions nor SuccessOrFailure for failures that should be handled locally. Exceptions are designed to be handled globally (at the top level) and SuccessOrFailure is designed to be simply a wrapper on top of them for cases where you need to work with multiple pending (non handled yet) exceptions at the same time. If you have just one exception at hand, then it should be always thrown.

P.S. If you write all your code in monadic style, that is a different case. Use an appropriate library (like Arrow) that has an integrated set of monads for all your needs.

ktkit commented 6 years ago

@elizarov Yes, I agree, and I got it. What I'm trying to highlight is the gist of the warning about the usage of SuccessOrFailure.
One thing to say: it SHOULD NOT be used ... with a few exceptions. And another thing to say: it SHOULD NOT be used in two cases ... or even CAN be used with two exceptions ... (which are the nullable type and the sealed class hierarchy).

When you view this type in the latter perspective, it doesn't look that unattractive. And it better corresponds its nature.

And by the way, Scala's Try type has the same usage issue, while the documentation doesn't say a word about it. So, even mentioning those two limitations is a step forward.

qwwdfsad commented 6 years ago

From the KEEP standpoint of view, it seems like SuccessOrFailure is a coroutines-specific primitive and any other usages are highly discouraged.

SuccessOrFailure is very similar to Try monad but lacks of monad combinators. And when something looks like a Try monad, it eventually will be [ab]used and sneaks into popular libraries, business logic etc. The fact that "real" Try monad is present in Arrow won't help, because it's not as common as stdlib and is not easily discoverable.

We already have Java 8 Optional experience: even with a tremendous educational effort we're still seeing stuff like equals written with optionals for "null comparison" or functional null-checks like Optional.ofNullable(value).ifPresent(value -> { log(value) }); (these are examples from popular github projects btw). I don't see how SuccessOrFailure may be different from the Optional pitfalls in its current state.

Unpopular opinion: If we don't want users to accidentally misuse SuccessOrFailure, we should move it to coroutines package and make all useful extensions internal, making it unusable standalone.