The issue with linguistic features

[emanjavacas]

• The analysis in terms of linguistic features is very interesting. However, most features are motivated from characteristic properties of rap lyrics. The exceptions are lexical diversity and word repetition. Interestingly, there have been earlier analyses of lexical diversity of rappers, showing that there are huge differences between them (see, for example, here: https://pudding.cool/projects/vocabulary/index.html). In a somewhat similar vein, repetition also occurs a lot in ‘real’ rap lyrics (Travis Scott ft Young Thug -- Yeah Yeah is just one example that springs to mind). It would be good to update the discussion of features accordingly. And what about other features that might be typical of neurally generated text, such as agreement mistakes and lack of global coherence? Would it be possible to integrate those?

[emanjavacas]

• I wouldn't agree with this "most features are motivated from characteristic properties of rap lyrics", what about morphology (to the extent that we model it) or syntax?
• While there are differences between human rappers, I imagine these are still not comparable to that between human and machine.
• It's true that the repetition score is mixing up both getting stuck and truly modeling rap-specific word repetitions...
• I am not sure what we should do on the issue of semantic coherence. I would still insist that 4 lines of rap would not necessarily show a lot of semantic coherence, even for real rappers.

[mikekestemont]

It's true that semantic coherence is an important criterion in itself, but as @emanjavacas says, I don't see us measure this adequately in snippets of this size. It's also very much the question whether people/rappers see this as an aesthetic criterion.

[fbkarsdorp]

• a lot of these "semantic coherence" measures are essentially what we do as well. I'd say we add something for future work (other and more complex linguistic features), but leave the analysis as is.

[fbkarsdorp]

RE repetition also occurs in authentic rap: I've added a footnote in the paragraph in objective feature importance:

Of course, authentic lyrics also employ repetition of words. However, the strong negative estimate suggests that the language generation models exaggerate this characteristic.

[fbkarsdorp]

To resolve the issue of semantic coherence, I added a line to the discussion:

The current study only covered a limited set of linguistic features. Future work should be directed toward understanding the effect of other, more high-level features, such as agreement of anaphora and semantic coherence.