Cabal file info not found / HttpExceptionRequest in exercises diamond and parallel-letter-frequency when running tests after submission

EspenBerget commented 2 years ago

The test passed on my machine, but when I submitted the code I got an error message:

Tests failed

Cabal file info not found for string-conversions-0.4.0.1@sha256:9c4075e54aa6cabfe244e07e2a1de720f3e98a0a6980fc671ab9c0da30f0f259,1617, updating
Selected mirror https://hackage.haskell.org/
Downloading root
HttpExceptionRequest Request {
  host                 = "hackage.haskell.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","Haskell pantry package")]
  path                 = "/root.json"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 0, addrAddress = <assumed to be undefined>, addrCanonName = <assumed to be undefined>}, host name: Just "hackage.haskell.org", service name: Just "443"): does not exist (Try again))

I search a bit and found this, which could be the same issue.

petertseng commented 2 years ago

Oh. It's probably because the test runner for security reasons is forbidden from having internet access. So something will have to change in https://github.com/exercism/haskell-test-runner, similar to how the Rust track had to do https://github.com/exercism/rust-test-runner/pull/27 for theirs.

degustaf commented 2 years ago

I just ran into the same issue. Should an issue be logged for the haskell-test-runner?

guygastineau commented 2 years ago

I had a similar issue. I also encountered an issue with Robot Name when I couldn't use unordered-containers for HashSet. I replaced it with Data.Set.Set from containers, and it passed the test-runner; so I assumed there would be some sort of whitelist.

petertseng commented 2 years ago

If I had to hazard a guess (but I have not tested this guess), I would guess that the test runner can currently use any packages that come with GHC, which would be those listed in https://downloads.haskell.org/~ghc/9.0.1/docs/html/libraries/index.html (or whatever version the test runner is using), which explains that containers can be used, but not unordered-containers, nor parallel, nor string-conversions.

petertseng commented 2 years ago

I notice that as of this writing there is no https://github.com/orgs/exercism/teams/haskell-test-runner (only organisational members can see that page, but as an organisational member I can confirm that it still does not exist). As of this writing there is only one person in https://github.com/exercism/haskell-test-runner/watchers and this person doesn't obviously seem to be in a ready position to solve this problem. So I don't suppose filing an issue in haskell-test-runner is likely to generate more solutions.

petertseng commented 2 years ago

A potential solution may make use of the extra-deps configuration option documented at https://docs.haskellstack.org/en/stable/yaml_configuration/#extra-deps. Using the similar strategy as https://github.com/exercism/rust-test-runner/pull/27, this solution would:

ahead of time, generate a list of commonly used packages. @ErikSchierboom, perhaps you could examine existing Haskell submissions and advise on the packages that are commonly used, so as to inform what should be in this list?
Download (for inclusion in the test runner's image) a version of those packages, according to that list. For example, https://hackage.haskell.org/package/parallel-3.2.2.0/parallel-3.2.2.0.tar.gz seems to be the format of the download paths.
- TBD: Does a potential solution support using multiple versions of a given package, or only one specific version? If it's only one specific version, we should be fine to use the version in the Stack resolver version that the track + test runner are currently using (efforts should be made to keep those Stack resolver versions in sync). If it's multiple versions, well of course then we'll need to figure out which versions of each package should be used.
Place every one of those packages into the extra-deps section in stack.yaml.

Those rough solution steps may need adjustments based on how Stack and Haskell packages work. I have no particular knowledge that would give me an advantage in either, so the steps may be wrong in some details, but the overall thrust of it seems to be in the right direction.

There may also be some value in seeing whether https://github.com/ndmitchell/offline-stack and/or https://github.com/AleXoundOS/haskell-stack-mirror-script are suited to this case, but I have not put in the time to examine them and verify whether they are.

ErikSchierboom commented 2 years ago

@petertseng Those sounds like very sensible steps. @iHiD could you send me a list of Haskell solutions? I'll extract the packages that are used from those.

ErikSchierboom commented 2 years ago

Thanks to @iHiD, I've been able to analyze all the Haskell solutions. As generally speaking people only submit the actual .hs file and not the package.yaml file, I had to look at the solution code. What I've done is look trying to find any import statements and then counting them. This leads to the following list:

Package	Occurences
Data.Char	26844
Data.List	19726
Data.Map	8466
Data.Maybe	5277
Data.Text	3559
Control.Monad	2846
Data.Set	2586
Data.Map.Strict	2468
Prelude	2398
Data.List.Split	2392
Control.Applicative	1223
Data.Bits	1144
Data.Time.Calendar	1093
Text.Printf	977
Data.Array	853
Data.Vector	847
Data.Foldable	816
Control.Arrow	762
Data.Function	762
System.Random	749
Data.IORef	566
Data.Time.Clock	456
Text.Read	417
Data.Time.Calendar.WeekDate	361
Data.Either	347
Data.Tuple	324
Test.QuickCheck	315
Debug.Trace	282
Test.Hspec	270
Test.Hspec.Runner	251
Data.Bool	243
Control.Lens	237
Data.Monoid	220
Data.Ord	219
Control.Monad.State	198
Text.Parsec	196
Data.String	192
Control.Concurrent.STM	191
Data.Tree	189
Data.MultiSet	180
Data.Functor	158
Safe	154
Control.Concurrent.MVar	145
Control.Parallel.Strategies	142
Data.Ix	133
Control.Concurrent	130
Data.Bifunctor	129
Data.IntMap	126
Data.IntMap.Strict	124
Data.List.Extra	120
Data.Sequence	113
Data.Attoparsec.Text	112
Data.Map.Lazy	108
Text.ParserCombinators.ReadP	104
Text.Regex.Posix	100
Data.Traversable	99
Data.Time	97
Data.String.Utils	95
Numeric	85
Data.Numbers.Primes	75
Text.Parsec.String	74
Data.List.Ordered	73
Control.DeepSeq	70
Data.Matrix	68
Text.Regex	67
Text.Megaparsec	64
Data.Digits	64
Text.ParserCombinators.Parsec	61
Data.Time.Calendar.OrdinalDate	57
Text.Regex.PCRE	57
Text.Megaparsec.Char	56
Data.Array.Unboxed	52
Data.HashSet	48
Prelude.Unicode	47
Data.HashMap.Strict	46
Control.Exception	44
Text.Megaparsec.Char.Lexer	44
Data.Either.Combinators	44
GHC.Unicode	43
Text.Parsec.Char	43
GHC.Exts	42
Data.Void	41
Data.Word	40
Data.Time.Calendar.MonthDay	38
Data.Time.Format	38
Data.IntSet	36
Text.Regex.TDFA	35
System.IO.Unsafe	35
Control.Monad.ST	35
Data.Time.Lens	34
Data.Composition	33
Control.Monad.State.Strict	33
Data.List.Index	33
Pangram	32
Safe.Exact	31
Control.Monad.Except	31
System.Locale	31
Control.Concurrent.STM.TVar	31
Criterion.Main	29
Data.Array.ST	29
Control.Monad.Writer	29
Data.List.NonEmpty	28
LeapYear	27
Control.Monad.STM	27
HelloWorld	27
Prelude.Unicode.SR	26
Text.Parsec.Text	26
Data.Semigroup	26
Data.Text.Read	26
Data.Strings	25
Data.Sort	25
Test.HUnit	24
Data.Functor.Identity	24
Data.ByteString.Char8	23
Data.List.Utils	23
Text.Parsec.Combinator	23
Flow	22
Data.List.Unique	22
Data.Text.Lazy	22
Control.Monad.State.Lazy	22
Data.Tuple.Extra	22
CorePrelude	22
Control.Monad.Loops	21
Control.Monad.Trans	21
Data.SortedList	20
Data.Array.IArray	20
Data.Vector.Unboxed	20
Control.Parallel	19
Parser	19
Data.Int	18
DNA	18
SpaceAge	17
Control.Monad.Reader	17
Control.Concurrent.STM.TMVar	17
Data.Time.Clock.POSIX	17
Data.Coerce	17
Criterion	16
Optics	16
Control.Conditional	15
GHC.Arr	15
Control.Monad.Trans.Maybe	15
Text.Parsec.Prim	15
Base	15
Data.Ratio	14
Data.Typeable	14
Control.Category	14
Data.IntMap.Lazy	14
Control.Monad.Random	14
Control.Monad.Combinators	14
Util	13
Control.Monad.Fix	13
Foreign.Marshal.Utils	13
Control.Monad.IO.Class	13
Lens.Micro.Platform	13
Acronym	13
GHC.Float	12
Text.Trifecta	12
Numeric.Natural	12
Control.Monad.Unicode	12
Matrix	12
Text.Parsec.Token	12
System.IO	11
Triangle	11
Data.List.HT	11
Text.Parsec.Language	11
Grains	11
Text.Casing	11
Card	11
System.Exit	10
Protolude	10
Data.Stream.Infinite	10
Data.Function.Pointless	10
Lens.Micro	10
Data.Graph	10
Types	10
Data.Containers.ListUtils	10
Control.Error.Util	9
SumOfMultiples	9
Data.Bool.HT	9
Control.Lens.TH	9
Text.Format	9
Data.Complex	9
Data.Vector.Unboxed.Mutable	9
Text.Megaparsec.Expr	9
Math.Geometry.Grid	9
Data.Map.Merge.Strict	9
Control.Monad.Combinators.Expr	9
Data.Char;	9
Bob	8
Control.Monad.Trans.State	8
Math.NumberTheory.ArithmeticFunctions	8
Control.Monad.Trans.Class	8
Control.Comonad	8
Math.NumberTheory.Powers.Squares	8
Data.Vector.Mutable	8
Control.Monad.Par	8
GHC.Conc	8
Data.STRef	8
Text.ParserCombinators.Parsec.Token	8
CollatzConjecture	8
Test.QuickCheck.Gen	8
Control.Monad.Identity	7
Data.MultiMap	7
School	7
Text.Numeral.Language.ENG	7
Control.Applicative.Unicode	7
Data.Array.Repa	7
Data.Array.MArray	7
Text.Show	7
Data.Attoparsec.Combinator	7
Data.Attoparsec.ByteString.Char8	7
Data.Attoparsec.Char8	7
GHC.Base	7
Data.HashMap.Lazy	7
Isogram	7
Data.String.HT	7
Data.Function.Tools	7
Locution	7
Data.String.Conversions	7
Data.Hashable	6
Control.Error	6
Data.Function.Memoize	6
PrimeFactors	6
Control.Monad.Instances	6
Data.Array.IO	6
Data.Dates	6
GHC.TypeLits	6
BST	6
TokenParser	6
Hamming	6
GHC.OldList	6
Language	6
Diamond	6
Anagram	6
Graph	6
Addition	6
MultiOperator	6
Math.NumberTheory.Primes.Factorisation	5
Data.Align	5
Control.Lens.Tuple	5
Text.Numeral.Grammar	5
Text.Megaparsec.Lexer	5
Genetics.Bases	5
Control.Concatenative	5
Data.Tuple.Select	5
Data.Data	5
Data.Modular	5
GHC.Float.RealFracMethods	5
System.Random.MWC	5
Data.Unique	5
Text.ParserCombinators.Parsec.Char	5
Data.Graph.Inductive	5
Data.Graph.Inductive.Query.DFS	5
Machine	5
Data.DateTime	5
Data.Fixed	5
Data.Bimap	5
Luhn	5
System.Time.Extra	5
Data.Bifoldable	5
Math.NumberTheory.Primes	5
GHC.Generics	5
ResistorColors	5
System.Random.Stateful	5
Hand	5
System.Environment	4
Control.Monad.Cont	4
Control.Monad.Trans.Either	4
Math.NumberTheory.Logarithms	4
Test.QuickCheck.All	4
Data.These	4
Formatting	4
Data.Either.Utils	4
Text.Megaparsec.String	4
Text.ParserCombinators.ReadPrec	4
Text.Read.Lex	4
Genetics.Aminoacids	4
Data.CaseInsensitive	4
GHC.Prim	4
Data.Proxy	4
"vector"	4
Data.Stack	4
Math.NumberTheory.Primes.Counting	4
WordCount	4
Data.DList	4
ClassyPrelude	4
Data.Text.ICU	4
Data.OrdPSQ	4
Sublist	4
Data.Tuple.Curry	4
Data.Attoparsec	4
Text.ParserCombinators.Parsec.Number	4
Text.ParserCombinators.Parsec.Combinator	4
WordProblem	4
Math.Geometry.GridMap	4
Math.Geometry.GridMap.Lazy	4
Data.List.Grouping	4
Data.Functor.Compose	4
Data.List.Safe	4
Data.Either.Extra	4
Strain	4
Data.Vector.Split	4
Test.QuickCheck.Function	4
Data.MonoTraversable	4
Optics.TH	4
Math.NumberTheory.Primes.Sieve	4
Data.Prefix.Units	4
Expr	4
Control.Lens.Operators	4
GHC.Real	4
Text.Regex.Base	3
Text.Regex.PCRE.Light	3
Text.Regex.Base.RegexLike	3
Data.ByteString	3
Text.Regex.PCRE.Heavy	3
Distribution.Simple.Utils	3
Text.Parsec.Error	3
Language.Haskell.TH	3
Data.Functor.Foldable	3
Foundation	3
Control.Lens.At	3
Control.Lens.Getter	3
Control.Lens.Iso	3
Control.Lens.Setter	3
Control.Monad.Writer.Lazy	3
Data.Default	3
IsbnVerifier	3
NanoParsec	3
Data.HashMap	3
GHC.Types	3
Data.Type.Equality	3
Data.Type.Bool	3
Criterion.Types	3
Frequency	3
Data.Global	3
Control.Monad.Extra	3
Control.Monad.Trans.State.Lazy	3
Control.Error.Safe	3
Data.Accessor.Basic	3
Text.ParserCombinators.Parsec.Prim	3
Math.Geometry.Grid.HexagonalInternal	3
Math.Geometry.Grid.Square	3
Control.Monad.Trans.Except	3
Data.PQueue.Prio.Min	3
Lens.Micro.TH	3
Control.Lens.Unsound	3
ProteinTranslation	3
Control.Applicative.Alternative	3
Relude	3
Text.RE.TDFA.String	3
GHC.Char	3
Squares	3
CryptoSquare	3
Text.Show.Pretty	3
Language.Haskell.HsColour	3
Language.Haskell.HsColour.Colourise	3
Control.Monad.Zip	3
Data.IntMultiSet	3
Data.Text.Manipulate	3
Control.Lens.Combinators	3
Test.DocTest	3
Data.Text.Lazy.Builder	3
Text.Regex.Applicative	3
Optics.State.Operators	3
Text.ParserCombinators.Parsec.Language	3
Control.Monad.Writer.Strict	3
GHC.List	3
Optics.Iso	3
Unsafe.Coerce	3
Control.Applicative.Lift	3
Data.List.Zip	3
Data.List.Predicate	3
Control.Monad.Trans.State.Strict	3
Clean	3
Control.Newtype.Generics	3
Data.String.Interpolate	3
Search	3
Text.Regex.Posix.Wrap	2
Data.Validation	2
Test.Framework	2
Test.Framework.Providers.QuickCheck2	2
ChessBoard	2
List	2
Data.List.Extras.Pair	2
Control.Exception.Base	2
Control.Monad.Fail	2
DsMonad	2
ETL	2
Data.FileEmbed	2
Control.Monad.Plus	2
Text.InterpolatedString.Perl6	2
Robot	2
Control.Monad.Trans.Reader	2
Genetics.Translation	2
Data.Vector.Algorithms.Intro	2
Data.Tuple.Utils	2
Data.Tuple.HT	2
Data.Time.Calendar.Julian	2
EnumQuoter	2
Control.Bool	2
Data.Vector.Lens	2
Data.Array.Repa.Repr.Vector	2
Data.Array.Repa.Index	2
Text.RawString.QQ	2
Data.MemoCombinators	2
Control.Monad.Primitive	2
Language.Words	2
Data.PQueue.Min	2
Sieve	2
Data.Functor.Classes	2
Statistics.Resampling.Bootstrap	2
Data.Universe.Helpers	2
Foreign.Marshal.Unsafe	2
PigLatin	2
Say	2
Data.Text.Unsafe	2
FingerTree	2
Data.Array.Repa.Stencil	2
Data.Array.Repa.Stencil.Dim2	2
Data.Sequence.Unicode	2
Language.Haskell.Codo	2
Data.Attoparsec.Text.Lazy	2
Pipes	2
Math.Geometry.Grid.Hexagonal	2
Control.Eff	2
Control.Eff.State.Strict	2
Control.Eff.Exception	2
Data.Strict.Maybe	2
Data.HashMap.Monoidal	2
System.Random.Shuffle	2
Pangram.Letters	2
Data.Text.Encoding	2
Data.Data.Lens	2
Control.Cond	2
Data.ByteString.Lazy.Char8	2
Relude.Nub	2
Safe.Foldable	2
Data.Set.Unicode	2
Cards	2
CollatzConjectureRecursive	2
CollatzConjectureHigherOrder	2
Counter	2
MyList	2
Data.Bits.Bitwise	2
Data.Char8	2
Weekday	2
PerfectNumbers	2
Clock	2
Data.Map.Ordered	2
Minesweeper	2
TwelveDays	2
RunLength	2
SecretHandshake	2
Test.QuickCheck.Arbitrary	2
LinkedList	2
House	2
Yacht	2
ListOps	2
Parsing	2
Meetup	2
Data.Boolean	2
Data.HList.HList	2
Data.HList.HCurry	2
Data.HList.FakePrelude	2
TextKind	2
Relude.List	2
Text.Numeral	2
Data.Functor.Const	2
Data.BinaryTree.Zipper	2
it?!	2
Data.Map:	2
Control.Carrier.Error.Either	2
Control.Carrier.State.Strict	2
Data.MemoTrie	2
TypedLogicTable	2
GHC.Natural	2
Distribution.Simple	1
System.IO.Error	1
System.Process	1
System.Console.GetOpt	1
Data.Thyme.Calendar	1
Control.Foldl	1
Text.Regex.Posix.ByteString	1
Debug.Hood.Observe	1
Network.CGI.Protocol	1
Data.Number.BigFloat	1
GrainsTH	1
Fastfoldable	1
maybe.	1
Text.Inflections	1
Data.HashTable.ST.Basic	1
Data.HashTable.Class	1
Control.Lens.Empty	1
Control.Monad.Trans.Writer	1
Regex.	1
Prelude.elem	1
GHC.Read	1
Data.Pair	1
Data.Graph.Inductive.Query.Monad	1
Data.Maybe.HT	1
Control.Monad.Trans.Writer.Lazy	1
Data.	1
Data.Time.CalendarTime	1
ONeillPrimes	1
Allergies	1
Data.Array.Repa.Slice	1
Data.Vector.Unicode.SR	1
Math.Combinatorics.Binomial	1
Math.Combinatorics.Exact.Binomial	1
Data.Eq	1
Data.MemoCombinators.Class	1
Prime	1
Data.Generics.Aliases	1
Data.Semigroup.Reducer	1
AI.HNN.FF.Network	1
Control.Monad.Trans.List	1
Data.Vector.Storable	1
System.Timeout	1
Control.Monad.Parallel	1
Control.Concurrent.QSem	1
Test.RandomStrings	1
System.CPUTime	1
PalindromeNumbers	1
System.Directory	1
GHC.IORef	1
Control.Concurrency	1
Database.SQLite.Simple	1
by	1
docs	1
modules	1
Data.List.Unicode	1
Data.Dequeue	1
System.Mem.Weak	1
Ersatz	1
Ersatz.Relation	1
Pipes.Attoparsec	1
Pipes.Lift	1
Pipes.Prelude	1
Pipes.Text	1
Data.Function.YaMemo	1
Text.Megaparsec.Text	1
Binary	1
Data.Time.Clock.TAI	1
Data.Convertible	1
Data.TIme	1
Hexadecimal	1
ReadShowRadix	1
GHC.Stack	1
Data.Decimal	1
EightDays	1
Test.Hspec.QuickCheck	1
Control.Lens.Each	1
Gauge.Main	1
Control.Newtype	1
Roman	1
Data.Time.Calendar.Week	1
Data.Set;	1
leads	1
Sgf	1
Math.Geometry.Grid.Octagonal	1
Data.List;	1
Scrabble	1
Formatting.Formatters	1
Data.NumberLength	1
this	1
ComplexNumbers	1
Data.Primitive.ByteArray	1
Text.Read.Extra	1
Alternative	1
RIO	1
RIO.List	1
RIO.Char	1
Data.Char--	1
Student	1
Grade	1
Data.Semiring	1
Data.Sequences	1
Data.Cased	1
House.Internal.VerseBank	1
Data.Graph.AStar	1
Extra.List	1
Text.Parsec.Expr	1
BasicPrelude	1
Lens.Micro.Extras	1
Data.Time.Calendar.Easter	1
Data.Profunctor	1
Data.List.GroupBy	1
Text.Scanf	1
Optics.Cons	1
Date	1
Genetics.Codons	1
Groupings	1
Data.Generics.Product	1
Data.Stream	1
Phone	1
Series	1
Beer	1
Data.Lens.Light	1
Data.Functor.Alt	1
StrainH99	1
StrainWriter	1
StrainPipes	1
Numeric.Extra	1
Data.Bitraversable	1
Data.Graph.Inductive.Graph	1
Data.Graph.Inductive.PatriciaTree	1
GHC.Integer	1
System.FilePath	1
If	1
Data.String.ToString	1
RIO.Map	1
RIO.Prelude	1
Data.Set.Ordered	1
NLP.Minimorph.English	1
Control.Lens.Cons	1
GHC.Conc.Sync	1
GHC.Enum	1
Classifier	1
Sentence	1
Data.UUID	1
Data.UUID.V4	1
Data.CharSet	1
Data.CharSet.Common	1
Math.NumberTheory.Roots	1
sData.Either	1
Control.Monad.Catch	1
Data.Char.Properties.Misc	1
Test.LeanCheck.Stats	1
Linear	1
GHC.Num	1
GHC.Show	1
Data.List.Match	1
Data.Map.Internal	1
Data.BinaryTree	1
OrdSet	1
HelperTypes	1
Control.Monad.Error	1
Text.RE.Replace	1
Text.RE.PCRE.String	1
Alphametics	1
Control.Monad.RWS.Strict	1
Cp	1
Nat	1
to	1
Control.Lens.Fold	1
Data.Set.Internal	1
Control.Lens.Traversal	1
Deque	1
Text.Builder	1
Data.Poly	1
TrileanLogicTable	1
Data.Time.Recurrence	1
Data.Text.ICU.Char	1
list.	1
Text.Regex.TDFA.Text	1
GHC.Maybe	1
Data.Kind	1
Math.Combinatorics.Multiset	1
GHC.Stats	1

Hopefully, we can use this to infer which packages to pre-load.

ErikSchierboom commented 2 years ago

@petertseng Of the above list, which packages should we be pre-loading in the test runner? We probably don't want to preload them all; for Rust, we've used the top 100 most used packages.

hraftery commented 2 years ago

FWIW, having completed 58 exercisms I found 3 (that I can recall - I can't search by "Failed") that expected additional libraries.

Lens Person. Description says "you may use any library you want [including none]". I picked the common lens but there are others that provided lenses, including those outside hackage.
Parallel Letter Frequency. Again, the exercise is not prescriptive on library use, but commonly one would use parallel.
Matrix. The source template provided uses Vector heavily so safe to assume almost everyone will want to use it.

Beyond that the list is endless. I found the restriction to bundled libraries useful for learning - modifying package.yaml is a lesson in itself. So I would only wish that the expected libraries above also be provided, and a gentle error be reported if you attempt to use more, but I can also see the appeal of trying to cast the net of "just works" a bit wider. Top 100 should satisfy many people, but then people might be coached into thinking any package is supported.

petertseng commented 2 years ago

I do not know of a good way to determine the package that contains a module. For example, Text.ParserCombinators.ReadP is in base, and Text.ParserCombinators.Parsec is in parsec.

If we want to use the top downloaded packages, we are fortunate in that https://hackage.haskell.org/packages/top provides a list.

Of course, a solution can start small.

For example, a first solution might only add:

QuickCheck (needed for dnd-character)
lens (in case someone doing lens-person wants to use it)
- perhaps other lenses libraries, but I wouldn't know. I didn't use one.
parallel (parallel-letter-frequency)
vector (matrix)
split (note 2392 uses of Data.List.Split)
random (robot-name, simple-cipher)
string-conversions (diamond - or we can see if we can rewrite the tests to not use string-conversions?)

petertseng commented 2 years ago

Question. How is it that hspec currently works in the test runner? That is certainly not in there by default.

ErikSchierboom commented 2 years ago

@petertseng The Dockerfile does a stack build on a pre-defined bit of code that has a dependency on the hspec library. See https://github.com/exercism/haskell-test-runner/blob/main/Dockerfile#L13-L14 and https://github.com/exercism/haskell-test-runner/blob/main/pre-compiled/package.yaml#L21.

I've created a PR to add the above-mentioned dependencies: https://github.com/exercism/haskell-test-runner/pull/32 Is listing them in the dependencies section of the package.yaml enough, or do the dependencies actually need to be used in the code too?

hraftery commented 2 years ago

I can confirm that my (local) test run of stack build --resolver lts-16.21 --test --no-run-tests with just the dependencies change (no source change) does indeed detect, download and install them.

FWIW, it also works if you put them under the top level dependencies: rather than the one under library:. Doing the former seems more self-explanatory to me, since the library: section is just a placeholder, copied from elsewhere. On the other hand, the intention is to make them available to the library, so it's not an illogical place to put them!

hraftery commented 2 years ago

Keep in mind that when the PR gets merged, the user will still be none-the-wiser if they happen to add a dependency which is not part of this list, resulting in the same error message in the OP.

ErikSchierboom commented 2 years ago

That is correct. This is something the test runner has to support.

ErikSchierboom commented 2 years ago

Fixed by https://github.com/exercism/haskell-test-runner/pull/32

hraftery commented 2 years ago

This is working for me now - previously failing solutions when re-submitted as a new iteration now pass. But I did forget to mention in my earlier comment: it takes a lot longer. Given all packages are installed each time, people will probably notice there's a step change with no obvious cause. In my experience it has gone from about 3 seconds to maybe 10 or more. That's actually pretty good (lens alone can take ages to download and compile) and quite reasonable as a new baseline, but one to be aware of.

ErikSchierboom commented 2 years ago

Given all packages are installed each time, people will probably notice there's a step change with no obvious cause.

Are they? The packages are downloaded and installed at build time, when the Docker container is built. It shouldn't affect the actual run time.

hraftery commented 2 years ago

Ah, my mistake. "build time" is not triggered every time.

petertseng commented 2 years ago

I must have forgotten to recommend the adding of text. We have a lot of exercises that recommend using the text package as an optional add-on, which is why I hadn't thought to look for it. We should add text, to support these following exercises.

$ git grep 'text.*to your list of dependencies' 
exercises/practice/acronym/.docs/instructions.append.md:- Add `- text` to your list of dependencies in package.yaml.
exercises/practice/atbash-cipher/.docs/instructions.append.md:- add `- text` to your list of dependencies in package.yaml.
exercises/practice/bob/.docs/instructions.append.md:- Add `- text` to your list of dependencies in package.yaml.
exercises/practice/diamond/.docs/instructions.append.md:- Add `- text` to your list of dependencies in package.yaml.
exercises/practice/etl/.docs/instructions.append.md:- Add `- text` to your list of dependencies in package.yaml.
exercises/practice/isogram/.docs/instructions.append.md:- Add `- text` to your list of dependencies in package.yaml.
exercises/practice/pig-latin/.docs/instructions.append.md:- Add `- text` to your list of dependencies in package.yaml.
exercises/practice/raindrops/.docs/instructions.append.md:- Add `- text` to your list of dependencies in package.yaml.
exercises/practice/resistor-color-trio/.docs/instructions.append.md:- Add `- text` to your list of dependencies in package.yaml.
exercises/practice/rotational-cipher/.docs/instructions.append.md:- add `- text` to your list of dependencies in package.yaml.
exercises/practice/twelve-days/.docs/instructions.append.md:- add `- text` to your list of dependencies in package.yaml.

adql commented 2 years ago

My (earned with blood!) solution to robot-name is still suffering rejection because of "missing" random. Which is very strange given the fix.

Is it maybe related to me starting the exercise at some time in the summer still on v2, and eventually solving it only today?

hraftery commented 2 years ago

Hmm, I concur. I opened the exercise for the first time and used the online editor. I get:

Could not load module ‘System.Random’ It is a member of the hidden package ‘random-1.1’. Perhaps you need to add ‘random’ to the build-depends in your .cabal file.

Perhaps the fix was only good for uninstalled libraries, not hidden ones.

adql commented 2 years ago

Yes, this is what I get as well. Then probably nothing to do with having downloaded it before.

In fact, this is similar what I got when submitting bob with Data.Text:

Could not load module ‘Data.Text’ It is a member of the hidden package ‘text-1.2.4.0’. Perhaps you need to add ‘text’ to the build-depends in your .cabal file.

Which seems different from the original bug that started this issue. Interestingly, text wasn't added to pre-compiled/package.yaml in https://github.com/exercism/haskell-test-runner/pull/32. Perhaps a new bug affecting packages regardless if they were preloaded or not?

iHiD commented 2 years ago

(cc @ErikSchierboom - Could you look into this one pls)

ErikSchierboom commented 2 years ago

@adql I think might indeed be a new bug. I've tried adding it to the pre-compiled/package.yaml file. I don't know how to fix this though. Any thoughts?

adql commented 2 years ago

@ErikSchierboom Not much of a clue, I'm not familiar with the machinery at the backend. Anyway, when I try the haskell-test-runner locally both bob and robot-name pass.

How much of the exercise directory is actually uploaded to the server when running exercism submit? Is it only the solution file or also others such as stack.yaml or <exercise>.cabal? I've noticed that the test runner is still set to --resolver lts-16.21, could it be related? (if only the solution source file is uploaded, then probably not).

ErikSchierboom commented 2 years ago

All files that are present in the exercise's directory in the track repo are checked out, with the user-submitted files overwriting any files.

petertseng commented 2 years ago

For robot-name, there are a few possibilities:

We could expect that the student should submit the package.yaml that includes random under dependencies, just like how we expect that if students use any dependencies they submit package.yaml
Or we could add random to https://github.com/exercism/haskell/blob/main/exercises/practice/robot-name/package.yaml, presuming that every student who wants to solve this exercise will use it (you might still get something that passes the tests without using random if you assign IDs sequentially, though you'd sort of be going against the spirit of the tests)

for those exercises using text, I see that text has now been added, so I am now hoping that if package.yaml is submitted, those will work.

adql commented 2 years ago

Ok, my bad for not having understood before that I should also submit package.yaml explicitly. The solutions to both robot-name and bob work when I do. Thanks for the help!

That said, it may be helpful to add an instruction to submit package.yaml where the exercise description suggests a library (as in bob); and/or a short comment about this under "Using packages" in Testing on the Haskell track. Otherwise the student might assume (as I did) that exercism submit xyz.hs also automatically submits project files.

petertseng commented 2 years ago

Absolutely correct to suggest this. https://github.com/exercism/haskell/issues/1038 and https://github.com/exercism/haskell/issues/1039 created

petertseng commented 2 years ago

Here is how future items should be handled:

If the error is Cabal file info not found for PACKAGE, decide whether it should be added to https://github.com/exercism/haskell-test-runner/blob/main/pre-compiled/package.yaml#L11, and if the decision is favourable, add it.

If the error is of the form:

Could not load module ‘MODULE’
It is a member of the hidden package ‘PACKAGE’.
Perhaps you need to add ‘PACKAGE’ to the build-depends in your .cabal file.

then the advice in https://github.com/exercism/haskell/issues/1006#issuecomment-1008054321 applies.

It's too much to keep talking about two different errors in the same issue, because then it's hard to tell what's relevant to what. Discussion about robot-name moved to https://github.com/exercism/haskell/issues/1042.

exercism / haskell

Cabal file info not found / HttpExceptionRequest in exercises diamond and parallel-letter-frequency when running tests after submission #1006